A Lesson from Wall Street

by Luke 5. August 2010 10:37

I recently had the chance to spend some time talking with the founder of a startup in the financial industry. He was looking for better ways to express the decisions he wanted to automate for a hedge fund management tool. And it struck me that the finance folks must know something that the engineers haven't caught on to yet.

Look at the Rules Fest speaker lineup and see all the finance folks. What do they know the we engineers don't? The cynic might say that they know the rules to skimming profit off a transaction. That seems to be the general feeling about high-frequency trading algorithms these days. But from my experience it's more than high-frequency trading - it's about automating all the little decisions.

The simple fact is that there is money to be made and in the financial industry it's easy to keep score. But when you look deeper, it's clear that beyond simple opportunism, the financial industry has learned how to automate the simple tasks. A large trade could be made any number of different ways. But rather than have people explore the myriad of options, computers do the heavy lifting. Doing it that way makes them more money. It's that simple.

That's a lesson we engineers need to absorb. The simple fact is that we need to get more efficient and we need to keep score if we're going to stay competitive. We've gotten good as a profession at using computers to automate calculations, and the next logical step is to use them to automate the simple decisions. That will make you a more efficient engineer, and efficiency is a great way to get more business.

Nobody Gets Paid to do Addition

by Luke 3. June 2010 03:02

There was a time when "computers" meant a small army of secretaries doing tedious math problems in the service of war or business. Today, nobody gets paid to do addition. Or multiplication. We have machines to do that.

For that matter, we don't use lookup tables for trigonometry or logarithm calculations. Machines do those, too. We don't manually calculate the standard deviation for a data set. We click a button in a spreadsheet to do that. We don't even have to type in the data since most of the time a machine collected it. The machine calculates integrals. It solves partial differential equations numerically. For almost any type of math an engineer could want to do, we're running out of problems that a computer can't solve.

Simple problem solving is the latest class of work to be automated. My GPS can route me around traffic jams. Netflix and Amazon can suggest movies and books you might like. Google made a fortune deciding what adds to show with your search. All of that problem solving was more than number crunching.

Today we use computers to manage supply chains, balance stock portfolios, and decrypt terrorist communications. We are so far beyond automating simple addition that it is amazing no one seems amazed. And this is a trend that is still accelerating. Be ready because the engineers that can harness the power of decisions in software will be the only ones able to manage and build on the complexity that's growing around us.

SpaceUp - Motivations

by Luke 1. March 2010 09:32

I've spent the past several months here on the Mindviews Labs blog, at conferences, and other places talking with people about automating engineering and how important I think it is. But I have to admit, as much as I enjoy working on this major project and solving the interesting problems that come up, this work really isn't my end goal.

I'm a huge space nerd. I think sending people and robots into space to explore and live is about the most exciting, fascinating, and interesting work in the world today. I got to spend a couple of days this past weekend at SpaceUp with a bunch of other people who are as taken with going to space as I am. If you're curious, you can see some of the Ignite talks that took place on Saturday evening at www.spacevidcast.com/spaceup/.

I think the only way living and working in space is ever going to be viable is if we can make it cheap, easy, and safe to design and build the rockets, satellites, space hotels, and mining stations we would be using. Right now the engineering is just too expensive.

The point is that the work going on here at Mindviews Labs is about making it easier - or maybe even making it possible - to do something that's important to you. We're building a powerful new tool. The same tool that can engineer my dream could also engineer yours. What would you build if you didn't have to worry about how much it would cost to design? Check in with us because your dream might be closer than you think.

Making Computers Make Decisions

by Luke 14. December 2009 12:11

The most valuable work that people do is make judgments. We decide what we want. We decide how to make it. We decide when we're happy with the result.

The problem we have today is complexity. The things that we want (from fancy cell phones to space satellites) require so many decisions that it takes huge teams of people to make sure they all get made. But many of those decisions aren't very interesting and, while necessary, don't add much value.

Any time I see a situation like that, my first thought is to make a computer make the decision. There are a number of approaches you can take. Decision tables and decision trees are straightforward tools that most engineers with some software skills can implement. Moving up the difficulty scale is implementing a rule-based system for your project. Beyond that, you'll be moving into the hard AI arena.

There are a lot of decisions that you can automate using the more basic tools and you should do some research and give those a try. Mostly I want to encourage you to think in terms of making decisions rather than crunching numbers. There are plenty of number-crunching tools but few decision-making tools. If you exhaust your options and are looking for other approaches, let me know and I can point you to some of our work or point you in the direction of some other experts that may be able to help you.

Think Beyond the Calculator

by Luke 24. November 2009 08:56

Part of my job is to help create great software. That's probably not a surprise to you. The other part of my job might be, though. That job is to help change our perspective (as engineers) on what software can do. The simple fact is that we've been using computers in our work almost exclusively as glorified calculators.

Excel? Calculator. Matlab? Fancy calculator. Super-computer time? Big parallel calculator. Code you wrote? Also probably a calculator. CAD/CAM software? Ah, now we're getting somewhere...

CAD/CAM software has offloaded the process of turning a part blueprint into a manufactured component from the machinist, who would interpret the drawing and devise a series of machining tasks, to the computer, which today can go from a CAD drawing to CNC instructions to finished part with little to no human intervention. But what about all the work that leads up to the CAD drawing?

We already have software that can make decisions about how to turn a design into a manufactured part. What are all of the decisions that need to be made to go from concept to that design? That's where we are focused. The future of engineering is creating tools that make decisions to help us, not simply building more powerful calculators.

Why Can We Automate Boring Engineering?

by Luke 17. November 2009 10:26

So why do we think we can automate boring engineering when we haven't seen anything like this before?

We have two - make that three - reasons for the claim. First, "boring" is a key part of why we can make progress. Creative tasks are usually engaging to people and keep them interested. It is the boring parts that are tedious and un-challenging. In short, boring work doesn't require a lot of creativity. And problems that don't need creativity are usually not as hard to solve with computers.

The second reason is social. Our challenges aren't just technical. Rather, we are working to see our tools used in engineering shops everywhere. Boring tasks are a bad way to introduce junior engineers to the work environment. Today's junior engineers are often skilled at computer programming and can make short work of the tasks that used to keep such team members busy until they had a chance for promotion. Instead, we can provide an opportunity for inexperienced engineers to learn from experience faster than they could have before. By automating the tedious work, young engineers can see more alternatives and more quickly understand the trade-offs that need to be made for a design to reach production. Our tools can change the way engineers learn on the job - instead of spending time crunching numbers and observing the work of others, they can spend that time exploring designs and understanding the strengths and weaknesses of them.

And the third reason that I almost forgot to mention was that we've already been able to use this approach in our case studies. For more information, contact us for one of our whitepapers that describes real-world scenarios where we've been able to use our tools. We think you'll be impressed by what we've been able to do.

ORF Reflections

by Luke 5. November 2009 07:14

As expected, October Rules Fest was excellent. For any other companies using or thinking of using rule based systems, I strongly recommend attending this conference next year. The focus is very much on real-world work being done in several industries using this technology. In a few months the conference papers should be released publicly, so check back and I'll try to provide a link.

One of my favorite talks was by Mark Proctor on the future of rule engines. His thoughts squared with many of our thoughts on rule-engine and language design, and we will be implementing several of his suggestions in the coming months to improve our systems.

There were also many other excellent presentations, but I'd like to see even more AI style expert-systems developers at the conference next year. Much of the use today is in algorithmic trading and processing by financial firms - the place where rule-engines seem to have found a home after the AI winter. But like so many technologies from that era, rule-based expert systems have quietly spread into a wide range of applications. We'd like to meet more of these developers and learn about what you have been up to.

ORF Coming Up

by Luke 23. October 2009 09:39

October Rules Fest 2009 is rapidly approaching and I'm putting the final touches on the papers and slides for my two talks. I will be giving presentations on "Graph Based Knowledge Bases and Rules" and "Designing a System of Rule Based Agents".

These papers cover some of our secret sauce that I mentioned in the last post. These technical innovations in rule-engine design/use have been key to our automation systems. These tools have given us the performance and scalability improvements we needed to make our software for automating boring engineering viable.

I believe many young technology companies are too secretive about their competitive advantage. Unless you've signed an NDA, the most you'll learn about their process or technology is that "the magic happens" and then you get the result. The software equivalent is "security through obscurity" which any security expert will tell you is little better than no security at all - in fact it often provides the illusion of security thus leading to overconfidence. So why is the analogy important?

We think hiding our secret sauce would work against us. We are working on hard problems with real technological innovations. The sharing of ideas and feedback will make our work better and help us build the tools we need to automate engineering. Our competitive advantage isn't in our secrets, it's in our execution. We're doing something hard. We're doing it well. And we challenge anyone to do it better.

Secret Sauce

by Luke 19. October 2009 06:36

Sometimes when I describe this company to someone for the first time, they ask me for the secret ingredient that makes our work possible. I get the impression that they are expecting one big breakthrough that I can point to, but I don't think that's the way most innovation works. So I have to disappoint them a little with my answer.

Instead of a single big secret ingredient, I think of it as several small innovations mixed together into a secret sauce. The recipe for us needed just four main ingredients: 1) better knowledge representation tools, 2) better knowledge capture methods, 3) modern expert systems, and 4) modern computing power. The first two ingredients are about managing complexity and the second two ingredients give us the performance we need.

Engineering problems have a key property that helps us manage complexity: many kinds of engineering decisions are not ambiguous. They can often be represented mathematically and that makes the job of teaching the computer to handle it much easier. But even simple engineering tasks can consist of dozens or hundreds of decisions that link together to make a solution and finish the work. The network of decisions on information creates complexity and that has always been one of the most difficult challenge for systems such as ours. We've created tools to manage the information complexity for our engineering domains and we've learned how to help human engineers teach computers what they know.

As for performance, much of that depends on how well we manage complexity and find ways to break large problems into smaller ones. Being able to run modern expert systems in parallel and over a network gives us the raw computing power we need to tackle many small problems at once. With good computation performance and a handle on complexity, we have the main tools we need to build automated engineering systems.


by Luke 12. October 2009 06:29

An "Aha!" moment is simply the last in a long series of steps that prepare you for when inspiration strikes. I want to share my "Aha!" moment with you - the one that led me down the automating engineering path that we're on today. Here is the spark:

Make software that performs work.

I was doing some research on expert systems and listening to a talk given by Larry Smith, an Economics Professor at the University of Waterloo, when I heard that message. To paraphrase his take on software opportunities, "We don't need more software to let us talk to each other. Make software that performs work." And I was struck by how true, yet unappreciated, that sentiment is.

The story of the internet billionaire warped the expectations of software developers everywhere. Google, Facebook, and now Twitter are all about communication and those are the imagination-grabbing examples that ambitious developers want to emulate. The job market for a software developer looks like a choice between the "glamorous" examples above and all of the "boring" line-of-business work going on within companies all over the world.

I agree that we have more than enough people working on communication tools. We've also spent the last two decades on the "boring" work of automating business processes (e.g., order processing, payroll, inventory management). But the third choice, the often overlooked choice, is to automate any of the other work that people do.

We can do amazing things by making computers perform engineering work. That was the realization I had when that final message clicked into place. That was my "Aha!" moment.

Mindviews Labs Blog

Sharing about the future of automated engineering.