The 21st and 22nd of October the Ruby and Rails 2010 conference took place in Amsterdam. About 250+ tech geeks, 15 speakers, 14o liters of coffee and a bunch of weird stuff that ended in a dance… All together a fun conference about the Ruby language and the Rails framework. Personally I had a great time in following the sessions. Listening to peoples ideas and crazy discoveries. And of course the networking with all the folks that came to the conference.
I would like to write a bit of a recap on the event. Write down some of the ideas I took from the sessions and the people I talked to and maybe go a little bit further with these ideas. First of all I would like to thank the organizers of the event. Great job!
Culture of Testing: Jon Yurek
Jon Yurek kicked us off with one of the most important messages in software development. YOU NEED TO TEST!
One of the major things Jon pointed out was that testing, or even version control, was not thought in school. I never realized that as I learned to test and use version control during my jobs I had when I was in school. He told us about him growing up with testing getters and setters in Java. Teaching himself to do useful unit testing and running into new problems when doing integration testing. Cucumber seems to be a very good framework for doing integration testing. Rspec is also a good way to do tests.
Everybody does testing. If it is just refreshing the browser or seeing if the code will compile is already a form of testing. Testing like that is not really sustainable… Tests are almost never about the technology of testing but it is almost always about the stories and about the way you test. Test Driven Development has proven to be a good way to do testing. Write tests first and write your code to match the requirements the test were written for. This is why cucumber and Rspec are such good ways of testing. They provide you a way to write tests if they were stories and requirements. It is harder to find out how and what to test then then to write the test. Using the scientific method comes close to the way you should be writing tests.
Greatest challenge to any thinker is stating the problem in a way that will allow a solution – Bertrand Russel
Tests do not only allow you to create stable code but also help with the sustainability of the code. Well written tests tell what a method should be doing. This in itself is a good way of documenting the function of the method. TDD helps you force yourself to write the tests and make them relevant.
My favorite quote of the talk was: “Projects that are not tested should receive public shaming”.
It was a very good talk and I would like to thank Jon for sharing his knowledge with us.
Git: the stupid NoSQL: Rick Olson
Up next was Rick Olson of Github to talk about how he used Git as a database and how it connects to Ruby. Although the idea that a version control system is a database seems far fetched it actually isn’t. Git is a document based system that stores data into a tree structure…. Sounds a lot like many NoSQL implementation to me!
How rick proposes one could use git is with a lot of the hidden Git commands. Not using the basics like Git clone, Git add or Git push that one would use when using it as a version control system. No use the underlying API’s that these basic commands use to actually store the file into a local repository. To do this in Ruby one could use the gem grit as a git wrapper. The proposed format in how you would store data would be something like JSON.
What was most interesting is that Git can also be seen as a graph database. As Git is a distributed system by nature it can have node to node behavior with different servers running their own version of the database and syncing whenever changes happen. A very nice idea for load distribution and optimization.
One of the examples Rick showed was the Gollum gem that was a very powerful wiki tool. Interesting how they wrapped a new API around it to create a wiki based on Git as storage!
I’m not going into detail about all the ways one could use Git in even more funky ways then one could imaging. It was a very high level talk with a lot of interesting concepts. I would to thank Rick for sharing his ideas.
Web-scraping techniques: Sebastian Burkhard
In the second room Sebastian Burkhard gave a talk about web-scraping with Ruby. Sebastian created a couple of aggregation websites for investments and for jobs. Sebastian uses information on other websites that do not have an API or something else to get the data he needs. Is this Illegal? not really as the data is already available for anyone to see. Might site owners get mad if you do take their information? maybe…
Sebastian explained how web-scraping works. Basically you use the structure of the page to find out what type of data is on the page and then you read it. The challenge is that every page is different. So for every page you need different selectors to find the correct data. Using the css, the xpath or a regexp you can point to a specific part of the page that contains the data. On a simple jobs page that might be the only H1 element that contains the job title. There are of course a lot more challenging examples but the basics are still the same.
What was interesting about Sebastian’s own approach that he did not use very specific selectors for every site. He uses a spider to crawl the page and he only select’s an area that contains the interesting information. From that uses that theory that every page is basically a set of labels and values. In 95% of the cases this seems to be true. Those labels and values can be turned into a hash.
Another problem is that when looking at the keys that they differ from site to site. Key mappers can solve this problem by connecting the names of fields and possible content. Now that data is usable for a new aggregation website. Keeping it up to date is just getting new versions of the data and comparing them to the existing data. Matching what is new, what changed and what has been removed.
Sebastian also introduced a scraping gem that he created and uses to do his scraping. He introduced and launched this gem as scraped_resource. It does spidering, content mapping, attribute mapping and has a build in normalizer. All the tools one could need to do scraping on the web.
An interesting question from the audience: “Would you be oke with other people scraping your website?”
Answer: “I would see no problem with that as it is already out there!”
Sebastian gave an interesting talk with some new insights for me. I would like to thank him for sharing those insights.
CouchDB: Mathias Meyer
CouchDB is a NoSQL document-oriented database and it’s hot! Mathias Meyer started with a nice introduction into the power of CouchDB. Then moved to how you can use it in Ruby and even in Rails.
Going too fast? It is complicated. Basically Mathias talked about a setup where one could do HTTP requests to get JSON files back. The URL would point to specific pre-determined selections of documents and parameters could be passed to refine those results. Major advantage is that it is BLAZING FAST! Other things like relations would need to build in views or at the end of the application. It’s not really supposed to be a relational database! Things like replication are actually implemented very well!
In Ruby using CouchDB is fine and with the use of CouchPotato it is very easy. CouchPotato hide the complicated API and presents it in very Rubyish way. Nice and clean! If you want to use Rails the gem SimplyStored is suggested as it provides an interface very similar to ActiveRecord.
Mathias has to do a wake-up session. As most of us were still half asleep before the first session. But with this interesting talk he woke everyone up in seconds (almost just as fast as CouchDB itself :-P). I would really like to thank Mathias for waking us up and of course his very nice talk about CouchDB.
Software Craftsmanship: Corey Haines
Corey Haines was by far the most inspiring talk of the conference. As a well mannered and well spoken speaker he graciously walked the stage to tell us about the art and craftsmanship of software. Using captivating examples about software development gone wrong and stories from the past he told why it was going wrong. What actually happens…
An example was shown of the Visa form for the USA. The form had an area where you needed to fill in If you have top secret information with you. When filling in the form it failed… Why does this happen? Was it not tested correctly?.. It wasn’t!.. Testing by clicking is alright but it is not very sustainable… Automated testing should be part of any developers repertoire. Automated testing is good, but using Test Driven Development is even better. Write your tests first and then make them pass. We all need to understand why this is and how we can tell everyone else that it is useful.
Corey: “I know a real rocket scientist, what he does is hard stuff. Really hard.”, “Business software and web apps aren’t hard but you have to pay attention to the detail!”
So if developing software is not that hard, why are we failing so many times? Well we all written our fair share of bad programs. The problem we face is that we just create software. We create but do not practice. Can a band do a nice live performance without any practice? Every time we start developing a piece of software we are doing a live performance, we tend to forget to practice. A lot of musicians use the technique of throwing away their music and just recreating their music over and over again to get the perfect sound. As a developer we can do the same by doing code Kata’s or creating prototypes. We tend to be afraid to trow away what we created because we think it is a waste of time. It’s not! In practice it’s all about balance.
Minimize the difference between “get it done” and “do it right”.
Software development is not new. It has been developing since the the second world war and we have grown and we have learned a lot. But a lot of the problem come from bad practices, not feeling like you have the time to figure things out and simple not knowing lessons from the past. Pair programming has been around since the 40’s. The women who were working on programs worked together on one machine to get a better result. Even things like MongoDB or CouchDB are new and hip! Their not really… Lotus notes was awful at email but had a document database.
Using techniques from the past and lessons learned is crucial to writing good code. Use the right design techniques, make the correct abstractions, use proper testing techniques, take care of your code, take care of your naming techniques. All help improve your code in structure, readability and performance.
Corey: “I can take a look at your code right now and tell you why it is bad code”
Take your influences from different places. Martial arts can provide you with techniques like Kata and Randori. Retreats are a by Corey advocated technique where you try to solve a problem within 45 minutes. This is too short so the pressure of finishing fades away. At the end of the 45 minutes you throw away everything and start over to solve the problem. Inspiration always comes from people around you so be open for it. Do not stick with the same crowd as the same people will give you the same ideas. Switch environments and inspiration sources to learn new things.
Corey: “Get paid to do things you enjoy”, “We have the greatest job in the world. Except maybe fighter pilots”
You perform well if you feel well. It all come down to happiness and what makes your job enjoyable. When you are happy about what you are doing you can convince people about what your doing that makes you happy.
Just a small recap: Test, Practice, Learn and be happy with what you are doing.
Corey made me happy and I would like to thank him for this incredible session!
Notable parts of the other sessions
The most notable part of the other sessions was by far the HTML5 dance that Justin Halsall prepared together with Sam Aaron (video). This was of course after a session about the new development in HTML5 and how we should use it.
Rubinius has always interested me from a distance. not too up close enough to get involved… yet. The notion that one push to the project would give you commit access was a nice idea though. It’s good to see open-source development at its best!
The talk about mountable apps in rails3 by Piots Sarnacki was something I was looking for. Fully functional rails applications you could plug into another base rails application. Will look into it but it seems to still be on the experimental side of the work.
The audio performance by Jeff Rose and Sam Aaron sparked a new interest in the work I’ve done with Max/MSP. But now as a proper programming language I will take a look at Clojure. Mybe even try to sell that to my media technology professors at the Leiden University.
- Test Test Test Test Test Test Test….
- NoSQL the Git or CouchDB way
- Ruby optimization and platforms
- Scraping the web with Ruby
- Getting better as a developer
- Just plain fun…
- and coffee…