Don't fly with Aires Airlines
If you happen to be neither Panamanian nor Colombian citizen and want to use their service between Panama City, Panama, and Cartagena, Colombia, be prepared to have an onward ticket. Aires is a small Colombian airline with mostly domestic service using small Dash 8 aircrafts. Their only two international destinations are Panama and Maracaibo in Venezuela, which was added recently.
Today we showed up on time to fly back to Panama City, our origin, and were refused boarding due to lack of an onward ticket. Arguing with their ground staff did not get us the boarding pass. They claimed each passenger who is not a legal resident of the destination country needs to have an onward ticket. According to them this were international law and each and every airline would work the same way.
So after a while we lost our patience and walked away to buy a one-way ticket from Copa Airlines, the largest Panamanian airline, without any questions asked. We got the boarding pass right away and everyone was very friendly and helpful.
Currently we are waiting here in the airport of Cartagena for the boarding process to commence. Apparently there is a free WiFi service, but I can't connect to it. So this post has to wait a few hours until we arrive in Panama City later today.
Taxis in Cartagena de Indias, Colombia
It's not the first time I visit Colombia. But this time a lot of things have changed. The city was heavily remodeled and there is still construction work going on at a lot of places.
Taxis in Colombia always have been yellow. In most Latin American countries they use cars with regular paint, so you have to spot the taxi sign on top of the car.
What has changed is that they now use only newer cars. A couple of years back some taxis were almost a moving wreck, but now everything is new they feature a big replica of the license plate on the doors of each side and even a bigger one the top. The one on the top is probably meant to be seen clearly from the air.
There are more changes visible in the streets. I'll try to take a few more pictures today before we leave tomorrow morning.
Don't trust your hosting provider
This weekend was quite eventful. At least on the negative side. Some may have wondered what happened to this blog. Why has it disappeared and why is there only a limited set of content available?
This blog has been hosted on two dedicated servers rented from a hosting company in Florida. The content was stored on a mirrored drive (not RAID-1, but an rsync mirror) and everything has pretty stable for a very long time. But there is always some new experience to be made...
At the beginning of this month I asked the provider for my two dedicated servers about the terms to cancel my current contract. As my Panama office has a 2 mbps dedicated circuit and it's running fine I was thinking along the lines of 'why pay for two dedicated servers far away, when you got everything right there in your own office'. The idea makes sense and moving to my own system would definitely save some money.
So my hosting provider in Florida - let's be fair and not bash him here in public - answered quite quickly. It wouldn't be any problem. I can get out of the contract without much hassle and it would be nice to vacate the machines till September 22nd. I should tell them when I were ready.
I wasn't ready and didn't gave them the green light to decommission the machines. One would expect that they simply extend the month-to-month contract and nothing would happen. Not in this case. Apparently the sales person had promised the very same servers to someone else and by September 22nd someone in their datacenter executed his orders and wiped the disks of my two boxes in order to allow a new customer to move in.
When I noticed the absence of my services this last Saturday I emailed them and a trouble ticket was opened. After a few hours without any response I decided to call them. But ... Where is their phone number. It turned out that they prefer email. After some digging I eventually found a phone number of their parent company and someone in India answered my call. Although this person was very nice he wasn't exactly helpful. It took until Sunday when finally a confirmation for what I've feared arrived. Both servers were decommissioned and redeployed for another customer.
Don't trust your hosting provider. When you rent dedicated servers you can create whatever complex and "secure" configuration as you like. A simple human error by a sales person can wipe out everything.
It was the first time for me that I've used a hosting provider instead of hosting the machine myself at the end of my own road to the Internet. I've grown at a time when the Internet consisted of networks and not big pipes between datacenters. My own server, my own UPS, my own line. Sounds complex, might be slow - depends on the load -, but it's my control, my decision and not somebody else's. So I'm back to hosting my own stuff on my own hardware at my end of the line. This time it's a virtual server running on top of a little AMD X2 64 powered HP Pavillion (quite a server :-)) in my office here in Panama. Nobody will fiddle with that setup again.
Unfortunately I have to leave in a few hours for a short trip and can't proceed rebuilding this site. But when I return I'll make it nice looking again. At least it was an opportunity to switch to the 2.0 release of the Pebble blog software.
SysAdmin services for small and medium-sized businesses
The number of dedicated servers at colocation facilities is large and seems to constantly increase. But who takes care of all those systems, who does administer them? When you rent a dedicated server, then you are the SysAdmin. The colocation provider gives you the box, the rack space, the environment and a powerful connection to the Internet. But as you are the only one who knows the root password, you are the SysAdmin. You can't rely on the technical expertise of the colocation provider.
Most contracts for dedicated servers make it clear what your responsibilities are. If your server is used as a Spam relay, as a source of attacks to other systems, as a repository for pirated software or whatever illegal or harmful activity, then you are held responsible and your system will most likely be cut off by the colocation facility.
And then there is the business reason why you rent a dedicated server. You do this, because the dedicated server provides a vital service for your business. If it doesn't operate smoothly, your business will loose money or at least reputation. So it should be your primary concern to make sure it's up and running at all times.
Most small and medium-sized businesses have a checklist when they go shopping for a dedicated server. They select their hosting provider carefully. And hosting providers make sure to convince their customers that reliable server hardware is a key element. So they offer brand-name servers, explain in great detail their own infrastructure (network, cooling systems, power distribution, generators, protection against natural disasters, etc.) and their sales people praise the technical expertise of their staff, which usually is real good - no doubt about that. Operating a large datacenter is a serious job and they better know what they are doing.
But unless you let them manage your dedicated server, all their expertise doesn't help you, because they don't have access to your system. And even if you arrange some case by case management solution, you can't expect them to have all the knowledge required to operate your key system as it would be required. They simply can't know all the specifics of your application, neither do they have the time to learn it.
As a small or medium-sized business you usually can't afford dedicated IT staff and tend to get along using computer science students as part-time admins or someone technically inclined does it on the side. If your small business develops in-house software, it's usually one of the developers who gets a second job as SysAdmin, which constantly takes him away from his actual duties.
For more than 20 years I've seen all this in small, medium and large companies. Large companies have their own IT department, their own datacenter and things usually work well. Small companies simply can't afford a professional SysAdmin and try to get along somehow. And medium-sized businesses always try to postpone the decision to employ a SysAdmin until some incident forces them to recognize there is a need for a qualified person to take care of their systems.
Does it need to be that way? I don't think so. It should be possible to offer a high-quality SysAdmin service for a price that is still lower than to employ someone with the required skills full-time.
Speaking of 'required skills'... That's another challenge. A real SysAdmin is not someone who can install Linux or has earned his MSCE degree. You need someone who understands and has actual experience in a great number of system and networking technologies. The number of persons who really understand TCP/IP including routing, firewalls, Unix systems, databases and at the same time know about the Software Development Life Cycle and can communicate with your developers is actually quite small. Either these persons are occupied with other clients or you would have to pay them more than would make sense economically for your company.
In our Panama office we've begun to develop a meaningful SysAdmin service at reasonable rates. As a service provided case by case simply can't work due to a lack of contingency, we've created package plans that allow us to schedule our resources, provide proactive service, do required maintenance work continuously and make sure we can respond when you most need us. Like in software development and other engineering disciplines SysAdmin tasks need to be well documented and there needs to be some kind of change management. So we use issue tracking tools like JIRA to make sure our staff and you as our client are aware of what's going on, who did it, when did it happen and whether there are related tasks that need to be taken care of as well. The idea is to get you involved, without taking you away from your core business.
There are still several details that need to be worked out with our clients, but we strive to provide a professional service in a very individual way while still being cost effective. So, if your small or medium-sized business uses dedicated servers at a colocation facility and you recognize some of the things I've mentioned above, why don't you get in touch with us?
iSCSI support in Linux
Linux iSCSI by Cisco joined forces with Open iSCSI. The result is code to be compiled against kernel 2.6.16 or newer.
Yesterday I successfully installed Ubuntu Edgy and upgraded to a kernel 2.6.17 compiled over night from source retrieved via git from the Ubuntu repository. The first boot with the freshly compiled kernel went well so I proceeded to download Open iSCSI from their Subversion repository and compiled it against the 2.6.17 kernel. After completing the configuration of the NetApp FAS 250 filer I could connect to it, map the LUN as /dev/sdb, run fdisk and mkfs and mount the new file system.
Actually it was easier than I expected it to be. Today I'll be working on support for an iSCSI based root file system.
Latin American business culture
Latin American culture is different than European or American culture. While in the United States everything is fast and people tend to work long hours, things in Latin America go by another pace. That doesn't mean Latinos are lazy, they do work hard, but they spend quite some time on social stuff like proper introductions at the beginning of a meeting and caring for personal relationships.
What you wear and how you speak determines how they treat you and how far you will get. Everybody in the office of a typical Latin company is properly dressed. Women wear perfect make-up and you won't find men without a tie and perfectly clean shoes - and those are not sneakers. It doesn't matter how important their work is, whether they receive clients in the office or are just working on internal stuff.
People address each-other with the proper title. A customer service technician named Carlos Rodriguez at the local cable company becomes Engineer Carlos. The higher the social rank more attention is paid to the right way of addressing a person or speaking about somebody, even if that person isn't present. In front of customers Carlos is always Engineer Carlos, because he had to study at the University in order to know how to install cable TV or how to use the provisioning system. His fellow technicians will call him just Carlos, if there is no customer around. But the receptionist, messenger or a sales guy will always refer to him as Engineer Carlos.
The social rank of a person depends on several factors such as your position in a company, your wealth, the name and rank of your family, whether you are a foreigner, your profession and on how you act and dress.
As a foreigner you are seen as a powerful person by default, because you need to posses some wealth in order to come to the country in the first place. Secondly, if you don't come as a tourist, you are seen as an important business person, as an investor and it's presumed you play an important role in the company you work for. While Latinos between themselves pay great attention to the clothes the other person wears, in some areas they do understand that foreigners might be important persons although they don't dress like a high-ranking executive.
How you are supposed to treat other people depends on the social rank of both parties involved. If you happen to talk to Engineer Carlos as the owner of a business, then you will address him simply as Carlos and never ever call him Engineer Carlos. You are the boss and Carlos is the guy who has to do something for you and Carlos has to respect you. It's important not to appear weak, but powerful. If Carlos has reason to believe, you might have the power to make his life hard, then he will work well. In those cases you don't ask for something, you demand it. But still with polite words and a regular tone, because you need to respect Carlos as well.
Another important detail is how you introduce yourself to other people and how your company introduces you. The Spanish language knows two different words for the English you. tu is used to address persons you know well or who are of similar or lower social rank as yourself. usted is more formal and generally used to express respect for the other person, but sometimes it's used as well to make clear that you are serious about something you are saying. Parents use usted talking to their children, if the boy or girl has done something wrong. The younger person addresses the older person with usted as well - even it's within the same family.
Let's say your name is Walter R. Smith. You are the owner of the company and have a meeting with a client. All your employees will address you and your client saying usted and refer to you or address you as Senior Smith - in front of the client. While in regular office communication they well use usted and you become Senior Walter, because the environment is less formal. On the other hand you will refer to your employee Roberto Rodriguez in front of the client as Senior Rodriguez and without the client's presence simply as Roberto. Of course you won't say usted, but tu, because he works for you and you don't have to respect him that much as he has to respect you.
Who Owns Your Computer?
If left to grow, these external control systems will fundamentally change your relationship with your computer. They will make your computer much less useful by letting corporations limit what you can do with it. They will make your computer much less reliable because you will no longer have control of what is running on your machine, what it does, and how the various software components interact. (By Bruce Schneier)
This will probably be true for hardware and software aimed at the consumer market which is very much dominated by the entertainment industry in all its variations. That entertainment industry not only consists of Hollywood film studios or music labels. It includes ventures like MySpace.com, Flickr, YouTube and other large scale web services that are not clearly aimed at paying professionals.
A consumer PC will probably no longer be the same grey box as those used in offices. There will be a distinction. The consumer PC will be a device to entertain the consumer and allow to use certain useful services to create user generated content. And maybe it will be the consumer PC that will not have a local copy of MS Office installed, but use a web based office suite. Maybe all those Web 2.0 companies already got it and are just waiting for the locked down devices to appear. Who knows?
Large corporations most certainly will be interested in buying trusted computers for their employees as well. As experience over the years has shown: PCs with a multi-purpose operating system that gives the user too much freedom can create a lot of trouble. Locked down workstations, with exactly that set of tools the employee needs, might be seen as a blessing from the point of view of management and IT department.
What is left are those devices used by developers and others. Open source software, free software, etc. won't go away. But it might well be that certain services on the web won't trust requests coming from such untrusted devices.
My work on a server farm with diskless blade servers
Some might ask what has happened to the work I was doing lately for a server farm with diskless blade servers. Everything is going well. Only there is nothing exciting to report for now. I've done some further experiments with Xen, diskless booting including the Linux Terminal Server Project and looked at other Linux distributions besides Debian/Ubuntu.
Currently I am waiting for additional equipment to arrive. The most important piece is a NetApp FAS 250 iSAN storage array with iSCSI support. Using an NFS root might be a nice solution, but probably an iSCSI root is nicer. ;-)
In about two weeks I can probably tell more about the project and whether iSCSI holds what I expect from it.
Virtual machines as crash test dummies to detect infections
This could be the most important milestone in the history of intrusion detection systems ever. FireEye announced a new appliance that uses virtual machines as crash test dummies to detect infected devices on a network:
FireEye technology uses virtual machines as a sort of crash test dummy for network security. Potentially damaging network traffic is sent into the virtual machines where the impact can be analyzed. If suspicious traffic is revealed, a responsible course of action can be taken to stop the network infection. No actual machines are harmed in this process and the damaged virtual machine is fully restored for reuse.
Although FireEye's website doesn't tell much yet about how it works Dark Reading, a ZD Net blog and Security Vibes provide some more information.
Apparently the appliance executes several virtual machines (VM) with instances of the Windows operating system and Windows applications used in the enterprise. Those VMs are then used as targets for the potentially harmful traffic. If a VM gets infected it should reveal suspicious network traffic and then the appliance takes action.
According to the information currently available the appliance is not in-band, but connected to a monitoring port on a switch to be able to see all packets passing through the switch. If harmful traffic is found the switch port it originated from is turned off.
Further the VMs resemble the different releases of operating systems the enterprise uses. That means one VM for each combination of Windows version (NT, Win 2000, XP) and service pack. That's quite a lot of VMs.
This sounds almost too good to be true. I can think of a lot of questions about how this really works and how it will scale. The idea by itself is brilliant. There is no doubt about that.
Some of my first thoughts about the implementation challenges were:
A 100 mbps link, if fully used, transmits about 750 MB per minute. That's slightly more than the content of a CD-ROM. On a typical DSL link with 256 kbps utilization it would still be 1.9 MB per minute.
The appliance has to send this amount of traffic simultaneously to all the VMs and check whether any of them shows strange outgoing traffic short after. If there are 10 VMs we are talking about 19 MB per minute that have to be processed in the case of the tiny DSL link. For the fully utilized 100 mbps link it would be 7.3 GB per minute.
Further the appliance needs to remember where the data came from. Without that knowledge it can't shut down the switch port of the infected device. Not every virus or worm starts sending packets right away. What if it waits an hour or a day?
As the appliance needs to wait for VMs to be become infected, it has to clean up and restart the infected VM in a clean state. That takes a moment. What will happen, if there are already several viruses or worms active in the network or new infections happen shortly after another? The load of the appliance and its memory requirements will increase. For every infected VM a fresh copy needs to be available in the background. I guess there will be a limit on how many infections can be detected or handled.
I'm not saying it's impossible to create such a device. But it is definitely a big challenge. And if FireEye has made it, then it will be a big success.
Tagging experiences made
Inspired by a post on Innovation Creators I have created a new tag Experiences and re-tagged some of my past articles. There is now a new entry in my sidebar as well. You can click on Experiences made to learn more about what new skills I learned, what work related experiences I made since I began to write this Blog. This is in addition to my work biography and the skills overview pages in my personal Wiki.
WiFi phone for Skype
As more and more WiFi hotspots appear all over the world it was only a question of when for the WiFi phone to be available. Now Netgear introduces the Netgear Skype WiFi Phone. You can use it to call anyone on Skype without a PC and via Skype-out regular phone number. It displays the contact list in the same way the Skype client on Windows or Mac OS X does. You can see who is online and select the contact to make the call.
It can be pre-ordered at the Skype store or Amazon. Here is the press release from Netgear.
That's quite an interesting move. It might be a very useful device for frequent travelers and be a blow to cell phone companies that bill horrendous roaming charges. This phone has now gotten on my watch list. Currently I use Vonage's softphone on my laptop and the Vonage service with a VoIP phone adapter in the office and at home. Calls between all the numbers in my account are free - just as Skype is free as well.
The only drawback, and it's a big one, is that this phone is not usable at hotspots that require you to visit a website first in order to use the hotspot. Unfortunately a lot of hotspots at hotels and airports are implemented that way.
Autonomous bot nets
Most certainly you have heard about bot networks that consist of thousands of PCs, infected by malware, that can be used for distributed attacks to bring down a server or to send out spam. All bots connect to a central server and listen for commands. That central server can be found and shut down in order to eliminate the threat.
Now this has changed. Published by the Internet Storm Center a new generation of bots is using peer to peer networking technology instead of relying on the vulnerable central server. Now bots can find each-other and pass commands along from bot to bot. Finding a few bots and shutting them down won't turn off the network. Instead it just doesn't matter.
This new development means that endpoint security has to be taken more seriously to reduce the number of infect-able systems.
The PC - An entertainment device
Just found this on Jeff Nolan's blog: Dell gives buyers the no-crap option
I bought my mom a HP desktop computer a while back and when I plugged it in for her I could not believe how many marketing icons were preloaded. I literally spent an hour cleaning it up figuring that the more stuff that was there the more confusion it would create. A few months later we had to send it back to HP because of a hard drive failure and went it (finally) came back… yep, all the preloaded crap was back with it.
Apparently PC manufactures think of their customer as a person who buys the machine as some kind of additional TV set, an entertainment device. People should consume, not use. They should buy, enjoy for a short moment, throw it away and buy the next fancy gadget. This sounds like an intriguing recipe for higher profits for the manufactures. But what about the long run? Doesn't that lead eventually to ever dropping margins and ever increasing product cycles?
Back in the 80's and early 90's "some dreamers" thought of the PC, and followed by the Internet, as devices that will help to increase people's knowledge. It appears that the manufacturers of those "entertainment PCs" know very well what the masses want. Looks like they aren't that much interested in gaining more knowledge but more fun instead.
The webtop - back in time?
What is all the hype about bringing classic desktop applications like word processing or spreadsheets to the web? CNN Money writes about The Webtop:
Software that was once the bailiwick of desktop computing is now going online. In fact, these web-based applications may someday entirely replace your desktop suite.
Ajax - or maybe we better call it DHTML as it was named earlier - is a useful extension technology that allows to build interactive webapps. There is no doubt about that. It just makes sense that changing a value in any input element on a page should have some effect on the state or values of other elements. Nobody will doubt it is useful to present the full contact information about selecting a person from a drop-down list.
But will anyone want to write books, business plans, letters, the important thesis online on somebody else's server? That's a bit awkward. People have shiny new, powerful desktop computers. There is a local printer attached, even a scanner or camera. And then they should login to a website running on a server at some far-away datacenter to write a letter? I don't believe that this is really a useful application.
Some proponents of online office applications might defend it saying that the user does no longer maintain her computer, doesn't have to worry about malware. It just sounds too good to be true. What hinders a virus to spread from a unsecure desktop operating system to jump to the server via the insecure web browser? That's no better than before. It's just another game for malware developers.
And what about data storage? Does everybody nowadays want to store their private documents on a server they don't control? What happened to people's wish for privacy? Maybe I will be able to store the documents locally. But then I can as well use a regular desktop application - can't I?
Server based applications are a great means for collaboration. A browser based user interface is great when I can't install software on my user's computers or when there are too many different devices. The web UI is perfect when my users use desktop PCs, tablets, PDAs or have to use computers at public places like an Internet cafe.
Once there was a promise: Write once, run anywhere. Do you remember? Java was said to be the technology that would make it become true. You would write a Java desktop application and your users can run it on every operating system that has a Java virtual machine (JVM). With Java Web Start there is even a technology that avoids shipping installation media to the user and perform a real software installation.
The only advantage webapps have is that they require absolutely nothing to get started. You just navigate your browser to the URL and can use them. But when I want to use a word processor or a spreadsheet that doesn't count much.
Or maybe we are now entering the age of the reborn mainframe with terminals? But probably to understand that one has to be older than 25 ...