Archive for the ‘Databases’ Category

No Rest For The Unemployed

Posted on January 26th, 2010 in Books, Computer Science, Databases, Linux, Mac, Open Source, Opinion, Programming, Ruby, Ruby On Rails, tips | Comments

After two years, yesterday was my final day of being a developer for BarterQuest. As anyone living in expensive New York City in the same situation, I have been focusing part of my energy looking for a new gig. I’m pretty confident, despite the current economy, that I will land on my feet sooner rather than later. I have a few leads, with one in particular that I am really hoping will come through.

In the meantime, I’m not just throwing my resumé in the face of companies everywhere in the U.S. I have also decided that since now that I have quite a few extra hours in my days, I should really do productive things instead of sitting on my couch and re-watching all previous seasons of 24 (believe me, I would watch all 7 seasons in a row if I could) or jamming away to Guitar Hero. This is a perfect chance to do lots of technical things I’ve been wanting to do for months, but just never got the time to do so. Here’s a short list of some goals I’d like to get started on.

Learning new stuff

For the past two years, I’ve been exclusively using Ruby and Ruby on Rails at my day job. I’ve always wanted to broaden my skills by doing other types of programming, but when you take into consideration that I would usually be at work more between 9-10 hours per day, plus a commute that would sap an additional two hours, there wasn’t much time for me to be able to do personal things, let along learn new stuff. Now that I’m finally free, I can now spend more time with those things I’ve wanted to experiment with.

I’ve always wanted to learn iPhone application development. I know the basics of Objective-C, and have the book iPhone SDK Development by Bill Dudney, but I was never able to sit down and code something up. I have a few ideas for apps, so even if I can make a simple app that’s accepted to Apple’s App Store will be an achievement for me.

I’ve been very interested in implementing Push technology to web apps, like Comet or Web Sockets, using nginx’s Push Module and Orbited.

Although I’ve never had the opportunity to work with extremely large data sets, I’ve always been curious about frameworks like Google’s MapReduce and Apache Hadoop, particularly how well they can “crunch” the data thrown at them.

Keep on with what I already know

As I mentioned, I’ve been using Ruby for years now, and I know Ruby on Rails and Sinatra pretty well. However, just because I want to learn new things doesn’t mean that I want to abandon this awesome language. In fact, I want to keep using it more with the latest toys.

Thanks to Ruby Version Manager, I was able to safely install the latest versions of Ruby 1.9 and MacRuby and start learning their new features. I was also able to check any possible compatibility issues in my older applications with different major Ruby versions. Seriously, if you are a Ruby developer using a Mac or Linux, install RVM now if you haven’t.

Recently, there have been more and more news about Ruby on Rails 3, the next major release for the wonderful framework. I’d like to stay one step ahead of the pack and start learning about the new changes before it officially hits the web. One of the leaders of the newest Rails changes, Yahuda Katz, has written lots of blog posts relating to the changes in Rails 3. They’re definitely worth a read.

Strengthening my shortcomings

There are quite a few things – development-wise – that have been bugging me for a long time, yet I’ve never taken the proper steps to correct. Now is as good a time as any to take on these things and finally conquer them.

My main weakness, as a web developer, is that I’m pretty bad at design. I know CSS and its properties, I know about browser incompatibilities (having been a victim many times before by the evil and immortal Internet Explorer 6 browser) and all that stuff. But as far as design goes, like font sizes, element placement, usability and colors, these things are not my strong suit. I’ve actually stocked up on some books about these subjects (like Don’t Make Me Think by Steve Krug and Web Design for Developers by Brian Hogan), so I hope that by the time my unemployment ends, I’ll be much better off making my work look good – or at least decent.

Another weakness I consider is that I get distracted from development from time to time. It’s not frequent enough to affect the quality of my work, but it’s enough to annoy me when I do it subconsciously and I then catch myself in the act. I have read some people who had some success using the Pomodoro technique, so starting tomorrow I decided that I’ll give it a try. There’s a nice little app called Concentrate for the Mac that seems to be just the thing I need during those times when I need to get stuff done and not get distracted.

Beef up my GitHub profile

I have to admit that I’m a little bit ashamed to see my GitHub profile virtually empty. For a long time, I’ve been wanting to add more of the projects that I have in my laptop to GitHub and see if some of them take a life of their own. Sadly, for whatever reason, I haven’t done that. Most of the times I’m a bit too critical at my code and think it’s embarrassing to make public, but that’s really what I need to do to get better as a developer. I can take criticism with the best of them, so there’s really no excuse. I need to make more of my code open-source, period.

Not only do I want to show my own work, I also want to give back to the community. I have used so many open-source projects over the years, yet I’ve only submitted a handful of patches to very few projects. I don’t want to be a person who takes, takes, takes and never gives anything back in return. So I’m going to take steps to correct that. I’ve started cloning some repositories of my favorite projects from GitHub to my computer to start reading their code more in-depth, which I had been doing anyway. I’ll check if these projects have Lighthouse pages with open tickets, or if there are any open issues on their GitHub page. A few years ago, Dr. Nic wrote an excellent post titled “8 steps for fixing other people’s code” that inspired me to start finding features or defects that I can handle.

I have to say, I’m only one day into this routine, and I don’t remember the last time I felt this free and liberated doing what I wanted to do. Full-time employment is great for earning money and making substantial stuff, but sometimes there’s a feeling of emptiness due to not being able to explore on your own. Being unemployed doesn’t mean that you need to spend all your time looking for work. Unless you’re truly struggling economically and can’t pay the bills in the next couple of weeks or even days, why not spend part of your time gearing up for the future?

MongoShort – URL Shortener using SInatra and MongoDB

Posted on January 14th, 2010 in Announcements, Databases, Open Source, Programming, Ruby | Comments

As many other developers, I’ve been experimenting with the different “NoSQL” data stores. Most of them are really great, like Redis, Tokyo Cabinet and CouchDB. My favorite so far, however, is MongoDB. I think it hits the sweet spot between a normal relational databases and key-value stores. After a while playing around MongoDB, I decided to do something with it. That something that I did is called MongoShort.

MongoShort is a very simple URL shortening service, written with Sinatra and MongoDB. I originally wrote this little app to be used as part of a larger Rails application. However, it was mostly done as a way to do something with MongoDB. Particularly, I wanted to start using the awesome MongoMapper library. It’s really a breeze to use all these tools together, and it makes developing small apps on Ruby really fun. I hope someone can grab a copy of MongoShort and learn a bit of the basics of these wonderful tools.

Go fork MongoShort on GitHub now!

Redis rocks!

Posted on November 5th, 2009 in Databases, Linux, Mac, Open Source, Opinion, Programming, Software | Comments

In my development blog, I previously wrote that MongoDB was the future of database storage. I might reconsider my decision, now that I’ve discovered Redis. In a nutshell, Redis is a key-value store. But it’s not a simple key-value database, as it has lots of commands and extra goodies, such as file persistence (so data can be stored from memory to a file, and restored in case of failure) and awesome data structures like lists (with push / pop) and sets (even ordered sets). Besides that, Redis boasts atomicity and even master-slave replication. It’s quite a lot of stuff for a lean package.

One of the things I love the most about Redis is it’s speed. Believe me, this little database is fast. Redis includes a benchmark tool, so I ran the default benchmark (Approximately 10,000 total requests per command, from 50 simultaneous clients) just to demonstrate its raw speed. These tests were run on a Mac Mini (1.83 GHz Intel Core 2 Duo, 2 GB RAM) under Ubuntu 9.10 Server:


dennmart@ubuntu:~$ redis-benchmark
====== SET ======
10005 requests completed in 0.14 seconds
50 parallel clients
3 bytes payload
keep alive: 1

34.89% <= 0 milliseconds
99.19% <= 1 milliseconds
100.00% <= 2 milliseconds
74111.11 requests per second

====== GET ======
10003 requests completed in 0.13 seconds
50 parallel clients
3 bytes payload
keep alive: 1

34.62% <= 0 milliseconds
99.79% <= 1 milliseconds
99.83% <= 2 milliseconds
99.87% <= 3 milliseconds
99.91% <= 4 milliseconds
99.94% <= 5 milliseconds
99.97% <= 6 milliseconds
99.99% <= 7 milliseconds
100.00% <= 8 milliseconds
74649.25 requests per second

====== INCR ======
10005 requests completed in 0.15 seconds
50 parallel clients
3 bytes payload
keep alive: 1

25.60% <= 0 milliseconds
99.73% <= 1 milliseconds
99.80% <= 2 milliseconds
99.83% <= 3 milliseconds
99.86% <= 4 milliseconds
99.89% <= 5 milliseconds
99.92% <= 6 milliseconds
99.95% <= 7 milliseconds
99.97% <= 8 milliseconds
100.00% <= 9 milliseconds
65392.16 requests per second

====== LPUSH ======
10001 requests completed in 0.13 seconds
50 parallel clients
3 bytes payload
keep alive: 1

37.62% <= 0 milliseconds
99.78% <= 1 milliseconds
99.82% <= 2 milliseconds
99.86% <= 3 milliseconds
99.89% <= 4 milliseconds
99.93% <= 5 milliseconds
99.96% <= 6 milliseconds
99.99% <= 7 milliseconds
100.00% <= 8 milliseconds
77527.13 requests per second

====== LPOP ======
10000 requests completed in 0.14 seconds
50 parallel clients
3 bytes payload
keep alive: 1

34.13% <= 0 milliseconds
99.69% <= 1 milliseconds
99.82% <= 2 milliseconds
99.86% <= 3 milliseconds
99.90% <= 4 milliseconds
99.93% <= 5 milliseconds
99.96% <= 6 milliseconds
99.98% <= 7 milliseconds
100.00% <= 8 milliseconds
74074.07 requests per second

====== PING ======
10000 requests completed in 0.12 seconds
50 parallel clients
3 bytes payload
keep alive: 1

43.79% <= 0 milliseconds
99.80% <= 1 milliseconds
99.85% <= 2 milliseconds
99.89% <= 3 milliseconds
99.91% <= 4 milliseconds
99.93% <= 5 milliseconds
99.97% <= 6 milliseconds
100.00% <= 7 milliseconds
86206.90 requests per second

Yeah, you read that right. On this somewhat low-end computer (by today's standards), Redis still manages to do over 74,000 SET (write) and GET (read) operations per second. Push and pop list operations also do well over 70,000 requests. On bigger hardware, expect these numbers to be much higher. Besides speed, I've also been impressed by the stability of Redis. I've been running Redis non-stop on the Mac Mini for over two weeks, and I've never had a single issue with it.

I got really interested in Redis after Defunkt (of GitHub fame) open-sourced Hurl. Hurl was an entry in this year's Rails Rumble competition. I used it quite extensively when developing with PayPal's APIs. After it was released, I noticed that Hurl used Redis for persistence. I was kind of puzzled why a key-store value was used for this. However, Defunkt wrote an excellent post mostly explaining the SORT operation, but also included some additional details on how Redis was used in Hurl. It made the decision to use Redis clear. Well, I'm also willing to bet that he used Redis because us programmers love to use the new hotness.

I feel that Redis will be something big soon. There are a few open-source projects using it, including the aforementioned Hurl and Resque. I already have a few ideas where I can use some of this functionality for current sites that I'm working on (such as simple counters, or a tag-like system where we can push and pop tags from a list). Check Redis out if you need some quick and simple storage.

What Have I Been Up To?

Posted on March 6th, 2009 in Books, Databases, Mac, Open Source, Opinion, Programming, Ruby On Rails | Comments

It’s been quite a while since I last updated this blog. I should seriously start writing more often. So, what have I been up to? With the cold weather here in New York City, it gets difficult sometimes to go out and have fun in the city. As usual, most of my time is spent on technology, software development, stuff like that. Here’s a brief recap of what I’ve been working on:

Got a new MacBook Pro, and I love it – I finally gave in and bought myself a shiny new MacBook Pro. I’m having a blast with it. For the open-source developer in me, this has everything I could ever want. It’s a beautiful and sleek machine that has gotten me highly motivated to start expanding my knowledge in different directions. While I still consider these laptops to be pricey, they’re really great.

Learning Objective-C for Mac and iPhone development – Since I got a nice Mac, I figured I should start learning Objective-C, then eventually get into Cocoa and the iPhone SDK. I’ve always been impressed by some nicely-done open-source Mac applications, and iPhone apps have always seemed to be so creative, so I wanted to learn how to build some myself. Since I’ve been working for so many years on dynamically typed languages (like PHP and Ruby), it’s kind of a total change of pace to go back to a somewhat-statically typed language like Objective-C. It’s been a bit of a challenge, since I last used a C-based language back in my college days. But it’s going along well so far.

I’ve compiled quite a few resources for learning all of these things. I recently purchased a book called Programming In Objective-C, which seems like the de-facto book on Objective-C, as it’s pretty extensive..And there are a few good screencasts – PeepCode developed a screencast dubbed Objective-C for Rubyists, and The Pragmatic Programmers have a screencast series by Bill Dudney called Coding in Objective-C. Of course, Apple has done a fine job with their Objective-C documentation. All of these resources should get you coding in Objective-C in no time.

The iPhone development part is mostly being learned because at my day job we would like to create a nice iPhone app for users of our site. I truly think that a mobile interface will expand our user base easily, even if people say they don’t really use mobile interfaces for a lot of things. I find myself using Facebook and Amazon’s iPhone apps more than I visit their sites.

BarterQuest is getting better and better – It’s been a wild ride on BarterQuest since we launched. We’ve been featured all over the place, from blogs to television shows, and we’re getting more and more users visiting, registering and trading on our site every single day. I’ve even traded twice already, and everything has gone as smoothly as it possibly could. It’s a great way to get the stuff you want by getting rid of the stuff you don’t want.

If you haven’t checked out our site yet, or haven’t done so in a while, I highly recommend you visit us soon. We’re going to be adding support for Real Estate in a few weeks, so our range of tradable items will greatly expand. And stay up to date with all that we’re doing by following us on Twitter.

Learning (and liking) CouchDB – My curiosity with CouchDB started when people were mentioning it quite a bit, and some Rails libraries like CouchRest were gaining some attention. So I checked it out for the first time, and I had no freakin’ clue why someone would use document-oriented databases, when relational databases did just fine.

Then a week I was working on a side project, and when trying to design my classes, I noticed that I would either have to denormalize my database tables, or be strict with normalization, but have a rather messy database schema. Then it just hit me – I finally knew the reason why CouchDB would rock in this scenario, where the data I was storing wouldn’t always follow the same structure. So I got into it, and I’m really liking it a whole lot. It’s something different, yet really useful in some situations. Seeing that there’s a market for databases like these, like Amazon SimpleDB, it seems like CouchDB (and document-oriented databases in general) will gain much popularity in the foreseeable future.

These are just a few things I’m playing along with. It seems like there are a lot of fun times ahead in software development and computing in general. I’m just happy to be able to ride the wave.

Securing MySQL’s Default Installation – Quick And Dirty Too!

Posted on October 2nd, 2008 in Databases, Open Source, Security | Comments

I just stumbled upon a script that MySQL includes nowadays called mysql_secure_installation. It runs a series of steps to remove some of the default installation options that MySQL installs, like an anonymous user and a test database. When you run the script, it does for the following:

  • Prompts you to change your MySQL root password, especially if you haven’t set a root password yet.
  • Removes anonymous access to your database server.
  • Restricts root access to localhost only.
  • Drops the ‘test’ table that’s installed automatically.
  • Flushes the privileges so that all changes are taken immediately without having to restart the database server.

Here’s the output when running the mysql_secure_installation in one of my servers.

In all, it’s something that I’ve been doing manually for a while. And I bet that there are many, many developers out there who, even on production servers, don’t even bother to remove these things. Obviously, this isn’t the “be-all and end-all” of MySQL security, but it’s a big step away from the default installation. Go run this script now! Well, unless you’re a security expert who already did this, either manually or with the script. If so, then I salute you.