This blog has moved
Over the last couple weeks, I’ve been working on a new blog. Among other things, the new blog should scale better and has a new cleaner template. Please update your links and RSS feeds.
This blog has moved, Update your bookmarks.
Stefan Rusek | This blog has moved
Stefan Rusek | LINQ Expressions as Fast Reflection Invoke
LINQ Expressions as Fast Reflection Invoke
Lately, I’ve been working on a side project that generates a lot of LINQ expression trees (LET). They provide a fun new way to dynamically generate executable code at runtime. Dynamic code generation has been around for a long time, and .NET has had dynamic assembly generation since the very beginning. The original mechanism, System.Reflection.Emit can be a lot of work to generate something small. (Not to mention it has little IL validation, so it will happily generate broken code that crashes your program with all kinds of strange exceptions.) Yesterday, I asked myself why not use expression trees instead of reflection for dynamic invocation. It immediately struck me as a wonderful idea.
The reason LETs are so great is that they compile down to real IL, and then get JITed, so they run at native speeds. Using reflection to dynamically invoke a method is far slower. Here is the code to create generic accessor delegates using both LET and reflection:
static Func<T, U> CreateWithLET<T, U>(string property)
{
var t = Expression.Parameter(typeof(T), “t”);
var prop = Expression.Property(t, property);
return (Func<T, U>)Expression.Lambda(prop, t).Compile();
}
static Func<T, U> CreateWithReflectionFast<T, U>(string property)
{
Type type = typeof(T);
object[] args = new object[0];
BindingFlags flags = BindingFlags.Instance | BindingFlags.Public;
PropertyInfo propinfo = type.GetProperty(property, flags);
MethodInfo getter = propinfo.GetGetMethod();
return (T t) => (U)getter.Invoke(t, args);
}
static Func<T, U> CreateWithReflectionSlow<T, U>(string property)
{
return (T t) => (U)typeof(T).InvokeMember(property,
BindingFlags.Instance | BindingFlags.GetProperty | BindingFlags.Public,
null, t, new object[0]);
}
I have focused on making the first two methods short, readable, and as efficient as possible. Both cases do as much work as possible in the create method. The third is to show that the reflection version can be written in one line, but if you plan to use it repeatedly, it will be much slower than the multi-line version above. This is a really simple yet very powerful example. It would be trivial to change this example to call a method and pass in arguments
I wrote four benchmarks. Each benchmark runs in a loop, the cost of the loop is subtracted from the total, and each delegate is given 100 warm up iterations to ignore the cost of just in time compilation. (I’ve included the benchmark driver at the bottom of this article.) The first benchmark calculates the amount of time it takes to create the delegates we are testing. The second benchmark times 100,000 iterations accessing an int property. The third benchmark times 100,000 iterations accessing a string property. The fourth benchmark times 100,000 iterations accessing two int properties and adding the results from within the generated LET .
I added the int tests because I thought the reflection tests would be adversely effected by needing to box and unbox the value of the property. In reality, reflection is so much slower that boxing accounts for about 2% of execution time.
| Create Delegate | Access int (100,000 times) |
Access string (100,000 times) |
Access 2 ints (100,000 times) |
|
| With LETs | 2.338ms | 0.45ms | 0.46ms | 0.53ms |
| With Fast Reflection | 0.46micros | 474ms | 462ms | 976ms |
| With Slow Reflection | 0.05micros | 988ms | 862ms | 1751ms |
I was surprised at just how much faster the LETs were than reflection. I expected them to be faster but they are about 1,000 times faster than the than fast reflection. The slow reflection approach takes twice as long as the fast reflection. I should point out that if you are only going to call the method once then reflection is still faster, but if you are going to use it in a loop, then LETs will generally win out. Another case were it makes sense to use the LET approach is when you want to precreate the accessor delegate and cache it so you can run it later when you need the speed.
Most of the 2ms it takes to compile the lambda expression is spent creating a dynamic assembly and various other things. So if you need to access multiple members dynamically and then do an operation on them, then LETs become even more appealing. It does not take significantly longer to to generate a lambda that access two properties than it does to access one. As you can see in the benchmark, a more complicated LET only takes a small amount of extra time, while using reflection is takes twice as long. This means that if you need to do some more complicated reflection, then LETs become more and more appealing.
Below is the benchmark function I wrote:
static void Run<T, U>(T t, Func<T, U> a)
{
// time loop
var sw1 = Stopwatch.StartNew();
int j;
for (int i = 0; i < 100000; i++)
j = i;
sw1.Stop();// warm up
for (int i = 0; i < 100; i++)
a(t);// run test
var sw2 = Stopwatch.StartNew();
for (int i = 0; i < 100000; i++)
a(t);
sw2.Stop();
Console.WriteLine(sw2.Elapsed - sw1.Elapsed);
}
Stefan Rusek | The 3 cast operators in C#
The 3 cast operators in C#
There are three cast operators in C#. The first is the traditional cast operator that uses parentheses, the second is the “is” operator, and the third is the “as” operator.
Normal Casts
This operator uses the same syntax as the original C cast operator. The key thing is that it always returns an instance of the type you are requesting. If it can’t do the cast, then it throws an exception. The majority of the time this is the desired behavior, because you want an exception earlier in your code rather than later.
object o = “hi”;
string s = (string)o; // works
double d = (double)o; // throws an exception since it can’t return a double
The “is” operator
This operator doesn’t actually return the result of the cast. It simply returns whether the cast would have succeeded. This is handy when you want to ask if an object is an instance of a type, but you don’t actually need an instance of the type. This operator is pretty useful, but it turns out that most of the time you really want to use the “as” operator, however, there are times when you really do want the is operator.
if (o is string) lblType.Text = “o is a string”;
if (o is double) lblType.Text = “o is a double”;
Some people might disagree that the “is” operator is a cast. They might say that it is a runtime type check or something like that, but type references in .net have the same value before and after a cast. This means that the normal cast simply does a runtime type check and if the result if false, then it throws an exception.
The “as” operator
The “as” operator returns an instance of the type request, but if it cannot do the cast, it will return null. This operator is handy for fixing the most common pattern for using the “is” operator. Since the “is” operator is a cast in disguise, when people use the “is” operator they usually end up doing two casts, and the “as” operator allows you to avoid the double cast.
if (o is string) {
string s = (string)o; // wrong because we are doing 2 casts
lblO.Text = s;
}
string s = o as string;
if (s != null) lblO.Text = s;
In C# v2, the as operator becomes even nicer with the null coalescing, because you can easily replace a non-matching cast with a desirable default.
lbl0.Text = o as string ?? “No value given”;
Common Mistakes
When I first learned about the “as” operator, I used it a bunch, and I ended up with code like this:
Converter c = x as Converter;
c.DoConversion();
The problem with the above code is that if the cast fails, then the second line throws an exception. If that cast fails, then you really want a cast excpetion and not a null exception, since the cast tells you a lot more information about the problem.
The other bad thing I did was before I learned about the “as” operator. I did this a lot:
if (x is Converter) ((Converter)x).DoConversion();
This doesn’t throw an exception, but it does do two casts. Casts are not as expensive in .net as they are in many other languages, but they are still non-trivial, so you want to avoid unneeded casts.
Stefan Rusek | Perfectly cooking dry pasta
Perfectly cooking dry pasta
Most directions for cooking pasta say something like this: 1. Bring water to a boil, 2. Add a pinch of salt, 3. Let it boil for 8-10 minutes. Occasionally, the directions might mention something about altitude and longer cooking times. These directions will generally get you from the point where pasta is dry and hard to the point where your pasta is no longer hard or dry, but it won’t get you perfect pasta. The directions below will get you perfect pasta anywhere in the world at any altitude with any water.
This is my quick guide to making perfect pasta every time.
- Bring water to a boil.
- Add salt*, you want to get the water to the point where it is just about the taste of sea water, generally around 2 tablespoons.
- Add the pasta and use the the technique below to determine when the pasta is done.
- Drain and add your sauce.
Don’t add any oil. Pasta that is over cooked tends to stick together much more than pasta that is cooked right. Your sauce will stick to your pasta better without oil.
Knowing when your pasta is done.
(click to see huge versions)
Dry pasta starts out hard and smaller than the final product. As it cooks, the pasta absorbs water, expands, and slightly changes color. The color change is the secret perfect pasta. At about 5 minutes take a piece of pasta out of the pot and bite it in half. The dry parts in the center look whitish. Check the pasta every couple minutes or so until the white has completely disappeared. As it disappears you want to check it a little more often. Once the white is just about gone, but not completely (see the picture above) you want to let the pasta boil for another 15 seconds. This is about how long it is going to take you to get out your strainer plus a few seconds. It is imperative that you strain your pasta as soon as possible.Your pasta is done! This means that if you are not quick you probably want to skip the 15 second wait. You must to serve it immediately. If you cannot serve it within 3 minutes, you probably want to leave it a little underdone.
* Many people think that you add salt to the water when cooking pasta in order to raise the boiling point of the water and make the water cook faster. The amount of salt needed to raise the boiling point of water enough to make to make a difference will also make your pasta taste awful. The real reason to add salt to the water is to provide some flavor to the pasta, since dry pasta has no salt in it. In fact back before we polluted the ocean so much, Italians who lived by the sea would use water straight from the sea to cook pasta.
Stefan Rusek | Don’t use IP geolocation to lock out users
Don’t use IP geolocation to lock out users
IP geolocation –using a user’s IP address to determine where in the world they are– seems like a good idea, but I am not convinced. I spend most of my time in Poland, but not all of it. I use a lot of online services, and some of these services work in Poland and some don’t. Certain media companies have made crazy licence agreements with certain other normally sane companies, and the crazy media companies want to prevent people outside of the US from consuming their content. (I understand why these companies have made these agreements; although, I don’t really think they are effective, but that is another blog post.) The majority of the normally sane companies use IP geolocation to determine if a user is inside the US. Up until recently most of these companies had an error message like the one below.
Most companies have recognized that it is retarded to explain just how bad their location detection technology is in your error messages (but in the last couple months most have changed their error messages). The problem with geolocation is that it is not really a reliable way to find out where someone is. Sure from a theoretical perspective it works, but in reality there are too many edge cases.
When I worked at Fog Creek software, we used a Canadian data center– or, I should say it was a Canadian company that has data centers in the US and Canada. We used their Manhattan data center. All our servers were in Manhattan, which is several hundred miles from the Canadian border. Since Pier 1 is a Canadian company, all our servers’ IP addresses looked like Canadian IPs. Joel gives more detail in an article about Pier 1. For the most part it was just funny. You type in http://www.google.com/ and you are taken to http://www.google.ca/, with all the text painstakingly translated into Canadian. However, you could forget about downloading something from VMWare’s website to one of our servers. Because the IP address was considered Canadian, many websites refused to recognize that we were actually in New York City. As the world gets smaller due to globalization, IP based location makes less and less sense. We will be running into more and more problems because people are in the “wrong” place when they aren’t.
For most service providers there is a much better solution. In fact, almost every website that uses an IP to block users already has the capability. I am talking about my billing address. Do you realize just how much work it is to get a billing address in another country? I have a billing address in the US, but I don’t yet have one here in Poland. Sure I have a mailing address, I am even renting an apartment; however, that doesn’t mean I can open a bank account here. I am waiting for my residency permit, and before that, I cannot even register with the government as really living at my address.
I have also seen the other side of this when Aneta and I tried to open a bank account in NYC. Aneta is not a US citizen. She has a green card now, but it was not yet approved when we moved to NYC. She and I went to the bank and opened a joint checking account. It took about 3 times longer than normal because Aneta wasn’t a US citizen and most people don’t understand what it means to be a pending green card applicant. (BCIS certainly doesn’t make it easy by not providing immigrants with some sort of useful documentation of their status, especially when they have to wait for 2 years by law for approval.) So After a lot of explaining and showing papers to the lady at the bank, we eventually got an account in both our names.
Another instance when I was struck by how hard it is to get an address was when we went to the DMV to get Aneta’s first driver’s license in the US. This was her first ID of any kind valid in the US. We had to provide tons of documentation just to show that she was legally in the country, then we had to show our marriage certificate in order for the DMV to let her use the same address as me. Typically, we don’t worry about these things. When I was 15 I went to the DMV showed my birth certificate, SSN card, and my mom showed her license. The hardest part was the written exam, which isn’t really that big a deal. Yet for immigrants it is not that simple, and for people who aren’t actually in country it is next to impossible.
For all their legendary reputation for usability, Apple can’t seem to make iTunes into an app I enjoy using. Yet, Apple has done one thing right. Maybe it is because the iTunes music store works in Europe where IP location just doesn’t make sense for anything. I can tell iTunes, that I live anywhere in the world, browse the library and look at all the fun French music that they can buy in France. If I try to buy some of that great French music, it will inform me that my US billing address cannot be used in the French Iiunes store. I can however buy all the music, movies, and TV shows I want in the US store, no matter where in the world I am. I realize that this system does serve me well, since I am not in the US most of the time, but many fewer people are in my situation, than are being wrongly being locked out of these services. It is also worth mentioning that I have VPN and Remote Desktop access anytime I want through the company I work for, so I can have a Seattle IP address whenever I need it.
Some companies like Apple totally get it right, while some companies get it half right. I can go to zuneoriginals.com in the US and pick out which Zune I want, which color I want, and what lithography I want. I can then order that Zune, and have shipped to an address in the US. If I go there from Poland, then I get the message you see at the beginning of this article. The site is pretty cool, and the lithography is something that Apple doesn’t offer. (I think it might boost the global desirability of Zunes if more people knew about this site.) The thing is that even if someone outside the US could get to the site, they could never get past the checkout page. Why use IP geolocation to block something that is already blocked? Luckily for me when I ordered a Zune last week, I just used Remote Desktop to connect to one of the icanhascheezburger.com servers and ordered it from a computer in Seattle. In a couple days it will arrive in Seattle and I will have to get someone to ship it here, but at least in a couple weeks I’ll be listening to audiobooks on my new 16gb Zune. Where Microsoft gets it right with the Zune is that I can subscribe to Zune Pass using a US address and get subscription music anywhere. I can’t buy Zune points though. I am sure they will get it all worked out if Zunes ever get popular enough for MS to sell them outside the US.
Aneta uses rhapsody, and we get mixed results with it. She can download music with the rhapsody software, but not stream it live from the rhapsody website. Amazon unbox is just broken. I can buy stuff, and I can play stuff, but if I try to download stuff, I am blocked. Hulu has change their error message from one like the one at the top, to a petition where you can register so they can try to get better licenses. (I guess Hulu wants the crazys to see how crazy they are.)
Of the companies I’ve mentioned in this article, Hulu is the only one that doesn’t require me to give them money to use their service. In their case I don’t propose that they drop geolocation especially since they are still only in the US, but they should provide a way to verify a billing address before blocking people.
IP Geolocation should be phased out wherever possible. Where it is not possible, it should not be the only means of making location based decisions about users. A US billing address is even harder to get than a US proxy server, and provides much stronger confirmation than an IP address can.
Stefan Rusek | Tańczymy
Tańczymy
Ja z Tatusiem i Anetą
Stefan Rusek | My last day at the Creek
My last day at the Creek
Wednesday was my last day at Fog Creek Software. Leaving is bittersweet, because I have many fond memories and great friends there. The developers are great friends and some of them I’ve worked with everyday for the past two and a half years.
For my last day, the company took all the developers out for a very nice lunch to send me off. We ate at Murano Ristorante. It was a lot of fun and it was a very nice treat. The restaurant was one that I discovered a couple months ago when I took a job applicant there for lunch. I highly recommend the homemade garlic potato chips! When we got back there was an awesome cake waiting. The cake was a cool cat shaped cake, along with six cheeseburger sh
aped cakes. The cheeseburger cakes looked surprisingly like real cheeseburgers, and when I took a bite, I was quite surprised by how tasty they were. The buns were made of muffin tops, and the patties were a lot like brownies. I was asked slice the cat, so I did the logical thing and cut the cat’s head off. Aneta ate the eyes when I brought the head home, and we both ate the brains!
My last two hours in the employ of Fog Creek were spent working with Brett to create sandboxes. I am not sure what more I should say. I suppose that I can say whatever I want, but I don’t want to spoil the surprise. I also don’t want to give anyone to expect sandboxes in future versions of FogBugz. The sandboxes that Brett and I made are still only prototypes, and it will be a while before they are used outside of the Creek.
The cat and cheeseburgers were to commemorate my new job. I started working full time for I Can Has Cheezburger on Thursday. I have been working a little part time for ICHC over the last few weeks already. So far I’ve been working on performance related work. The next couple months should be real exciting, Aneta and I have a little more than 3 weeks left in New York. Then we drive down to Tennessee, a few days later I fly to Seattle to work on-site at ICHC for a month. Then back to Tennessee for a few weeks, and then to Poland!
Stefan Rusek | The start of something new
The start of something new
For the past two and a half years, I have worked at Fog Creek Software. In that time, I have worked on some awesome stuff, and I have made some great friends. Recently, my wife and I decided to move back to Poland. My wife is originally from there, and I lived there when we met. Unfortunately, Fog Creek Software requires all employees to work on-site in New York City. This means that even though I remain a big fan of Fog Creek Software, I had to start looking for a new place to work.
So I started the job search. Eventually I stumbled upon a job that sound like a ton of fun. The job description didn’t exactly specify that I would be able to work off site, but I though I would give it a try. Not too long there after, I accepted an offer from Pet Holding Inc. What! You’ve never heard of PHI? Oh, I guess you might have been to http://www.icanhascheezburger.com/. I am pretty stoked about my new job. It is a much smaller company than the Creek. (Fog Creek is still a small company, though it has grown to 4x the size it was when I started.)
Stefan Rusek | My Bank Rocks!
My Bank Rocks!
Sometime in the last week or so, someone was able to get my bank information. I am not sure how or when. and it isn’t really the point of this post. The cool thing is how my bank handled it. As soon as I noticed two transactions for a thousand dollars each, I called up my bank. With in minutes, I was on the phone with a security agent. The agent was courteous and helpful. He said they would send us new ATM cards and the bank would refund the money. Later that day two more transactions went though that had been pending and the total came out to three thousand dollars. At this point the bank did exactly as much as you would expect it to. I really appreciate how quick and helpful they were. The real kicker is that they not only refunded the 3,000 dollars and 6 dollars ATM fees, but they refunded 0.02 cents in lost interest. I’ve circled the interest below:
I don’t get any referral money, but if you are interested HSBC Rocks!
Stefan Rusek | Aneta is Blogging
Aneta is Blogging
Aneta has started blogging, you should check it out. She has some awesome photos and interesting things to say.
Go and comment!