CAMP Systems
I work for CAMP Systems which is (I believe) the industry leader in providing software and information for keeping mid-sized airplanes (think: corporate jets) in good working order that meets regulations around the world. The product does what it's supposed to do, and I'm sure it helps a lot of people keep up with regulations and keep their planes maintained.It's also expanding to help people find the nearest parts for repairs (which I have been told is a pain point with planes currently), and we keep buying more companies to expand the services we offer. The business model is pretty good, and as long as we keep customers at least moderately happy, CAMP will keep on existing as a "good company" from the perspective of money making.
Good for upper management. Bad for low level employees.
The focus has been entirely on developing business relations at a break neck pace. And from the point of view of upper management everything is going exceedingly well. But there's a problem. The flagship application is rotten to the core, and working in it is an exercise in misery and frustration.
The Flagship Product
The application is called CAMP 3.0, and it is the interface for basically everything we offer. The older version of it is something we call "CAMP Classic", which I guess spanned version 1.0 and 2.0. I think about 6 years ago (maybe more, but not much less) CAMP decided to re-write the application with new technology to make it better. CAMP Classic is written with VB6 services, classic ASP, HTML, CSS, and JavaScript. That's fine because it was written over 15 years ago when VB6 and ASP were reasonable at that time. And it makes sense that we'd want to re-write the application to bring it into the present. Hooray... CAMP made a good decision.But it failed to execute well. The current working application today REQUIRES the classic version of the application to function. It still relies on the old VB6 services, and even still uses a number of the ASPs right in the application. We just link right to the old pages in IFrames. So today as I write this, the flagship application for this company that's doing well depends on 15+ year old technology.
The next bad thing about this is that we still support the old version. Why? Because some of our customers don't want to use the new application. The new one has possibly the worst designed interface of any professional application I've ever seen. I'm a software engineer who uses computers and smart phones in my personal life. I'm around interfaces all the time. I've seen a lot of different user interfaces, and if I were to consider using CAMP for maintaining my planes (if I had planes), I'd look at the interface and wonder what the heck is going on. It's a terrible interface. I'm not surprised we have clients that refuse to use the new version.
So we as developers have to maintain the old application, and we have to maintain the new poorly written one that is sitting on top of the stupidly old code. But I'll get more into detail about how poorly written the new code is later. For now I'm trying to stick with stuff that is obvious to a customer using the product. What I've covered so far is that we have a really old version of the application that is obviously designed in the 90s (I guess I pointed out that the languages used are old which isn't obvious to the customer), and that the new version has a terrible interface design.
The next thing customers would notice is that the application is slow. I don't know about you, but when I am on a site like amazon.com or one of google's products, when I click on a link, I expect the page to load in less than a second. I accept that sometimes the internet is busy... or a server is busy... or even my own computer is busy and going a little slow, so a page might load in 3 seconds at worst. But if every time I navigate to another page it takes at least 3 seconds... and it can sometimes take 10, 20 or 30 seconds... I'm going to be super frustrated. If I was shopping online for something and it took 30 seconds to load a product page, I would have closed the page after 10 seconds and gone somewhere else for the shopping.
An Effort to Make Things Better?
CAMP recently moved our database server to a much beefier set up, so some of the slowness is in better shape... but... the slowness is because the software is written so poorly. It's always going to be slow even if we have the best hardware available to serve up our application. And it surprises me that customers don't complain more about the slowness. Maybe the people using the application are more patient than me. Maybe there's not a better alternative.That takes me to my next point. CAMP is doing as well as it is not because the product is good and people want to use it. It's doing well because CAMP has relationships with aircraft and engine manufacturers that include a year of using our product for free. In that time we build up a huge amount of data about that aircraft that is not easy to get out of our system and into another one. Our customers stay with us only partly because our competitors don't offer as much (primarily around keeping up with government regulations), and mostly because moving away from us would be so difficult it's basically impossible. Our business model is to trap customers. And it seems to work. CAMP is still in business.
I'm obviously not a business man. I don't know how to run a business. But if I did run a business, I would want to provide a product that my customers choose to use because it's a great product. I want to stay in business because my customers are happy with what I provide for them... not just that they accept it, and certainly not because they feel trapped. That's awful. I should point out though that I don't have direct contact with customers. I don't know for sure how happy or unhappy they might be with what CAMP gives them. All I can tell you is that the application is objectively bad when compared to other professional service applications, and that I think I would be pissed off if I was stuck with our application.
Why is it Awful to Work Here?
So now on to why this makes CAMP a terrible place to work. If you're a software engineer, you probably already know, but I'll clarify and provide more detail, as well as mention what I think upper management is doing wrong.Application Foundation
The foundation of the whole application (as well as some of our side applications) is the database. This is true of web applications in general. They almost all need a database to store information. There's an incredible amount of information and knowledge about how to design a database so that it functions well depending on your needs. Not only was the database designed poorly for CAMP's purposes, but there are mistakes in the implementation that shouldn't exist in any application's foundation including a ridiculous amount of overlap of data, and tables broken apart for no good reason at all. The naming of tables and procedures is terrible, and there's zero documentation to help an engineer figure out where anything is. And the database is somehow over 3 terabytes in size. That's over 3000 gigabytes. And it's ridiculous. The database could be much smaller if it were designed properly, and without losing any data. This is the repetition of data I was referring to. I could only provide an estimate though about how much smaller it could be. The number of tables in is in the hundreds, and I've seen new tables made because we don't know what's already available... It's just an awful foundation, and it's MISERABLE to work in.Oh right... I didn't even remember to mention yet that the most used stored procedure is over 4000 lines long. Four thousand. I'm pretty sure that's bad programming right there. And how do people maintain a beast like that? I can tell you it isn't fun.
It's like building a house a really. The database is supposed to be a cement object that goes at the bottom of a house... to give the house a solid thing to rest upon. But instead of a well laid out cement foundation for a house we have a pile of rocks arranged to look sort of like a foundation. Asking a builder of houses to build a house on the pile of rocks would likely get a look like you're crazy. But CAMP went ahead and built the house which now will forever rest on the pile of rocks that comes with all kinds of problems, including make additions or fixes difficult to do.
Data Access Layer
The next layer that you're supposed to have when writing software is a data access layer. This layer is basically a middle man between the database and the application you're writing. It's sort of like at a store. Most stores organize their products so people can find them easily, and provide you ways of finding those products. But there are warehouse stores too where things up on shelves out of reach, and finding what you're looking for requires more effort, and you might need help getting what you want from wherever it's stored. The data access layer is the public face of the database meant to help your application to get what it needs.It also means that if the database changes, you can just update the data access layer to deal with the change, and most likely you won't need to change anything else. The application doesn't care how the database stores its information... it just looks to the data access layer to give it what it needs. That's it. Really simple and good concept.
I explained all that to tell you that CAMP has an awful data access layer. It's handled with four different technologies and in a variety of places. You have to know where to look when you need to make a change. And sometimes you need an old VB6 app that someone wrote to re-build an object for you.
There are world class excellent pieces of software available that can build a data access layer for you that we SHOULD be using, but we don't. Examples include Entity Framework and NHibernate. Instead we have home grown VB6 apps (more than one) that generate data access layer classes, a web API that uses things like TT files, inline SQL, and LINQ to SQL. That last one (LINQ to SQL) is potentially really good, but we only use it randomly instead of consistently. We don't actually have a data access layer. Instead of a nice store organized well to help you find what you need, we have a junk yard that we have to search through to find what we need. It's awful to work in.
Business Layer
The Business Layer is the next layer up the chain. It's job is to handle all the thinking. It provides the place where the logic is supposed to go. It decides what to ask for from the data access layer, and gives the information to the next layer (the user interface layer). It also is supposed to handle business logic.Let's say that you have an online store, and a customer clicks on a link that takes them to product details. The business layer is what understands that the user clicked on a link for product details, and knows what to ask the data access layer for. It knows to ask for description, price, customer reviews, and to look for possible deals or promotions that the product is part of so it can give that information back to the user interface.
It's where all the decisions about what to do based on what the customer asked for happen.
Obivously (because I'm explaining it), CAMP doesn't do the business layer right. In fact... CAMP doesn't really have a business layer. It's kind of a pass through. It seems like CAMP puts the business logic in the database (where it absolutely should not be) or in the User Interface (where it absolutely should not be). That huge 4000 line stored procedure I mentioned includes calculations that should be in the business layer. If we need to change some math, we have to update the database. If we need to add a new calculated field, we update the database. We should be doing the calculations in the business layer. The data base should just be concerned with storing data and returning the stored data quickly. The business layer should decide what to do with the data and perform any calculations that are needed.
But that's not what we do. We have convoluted calculations done in a language that isn't meant for business logic, and we are stuck maintaining the code this way. It's part of the miserable experience working here. And it is one of the major contributors to bad application performance.
User Interface (UI)
CAMP's interface for it's flagship application is awful in every possible way. Look at it as a customer and you see possibly the least intuitive interface you've ever seen crammed with so much text and graphics baubles that you have no idea what you're supposed to do. And even when you start using it, it remains crammed with so much text and graphics baubles that it's like looking down a sketchy strip of road with dozens of neon signs all trying to get your attention.But for a developer, the reason it's awful to work in is different. The entire UI is written using ExtJS which is a JavaScript framework. A framework is supposed to make writing code easier by doing some of the heavy lifting for you. It gives you pre-created things like grids that let a user re-sort columns, re-order columns, add/remove columns, and so on. Those are admittedly hard to write yourself, so it's easy to understand why companies would choose to use a framework or a library.
But ExtJS is awful. It's flat out awful. The benefits it brings are far outweighed by the problems it brings. I've written other blog posts about that, so I won't go into deep detail here. I'll just say it's awful, and then point out again that the whole UI is written with this pile of crap. So for a developer here at CAMP, if you want to change the UI or fix a problem with it, you have to use this arcane third party tool to do it. And it fights you every step of the way. It makes simple tasks that would take 5 minutes take two days. Literally. I'm not exaggerating. Everything you want to accomplish in the UI takes a lot longer because of ExtJS.
I have to give fair due here though. CAMP is stuck with an earlier version of ExtJS. Most of the application is written in 3.2 where I think ExtJS is up to 5 or even 6 now. It might be okay now, but I find myself doubting it. I just wanted to be fair. So there's that.
But regardless of what ExtJS might be doing now: CAMP is stuck with the old crappy version, and we have to work in it. Working in the UI is an exercise in misery and frustration.
