Wednesday, October 29, 2014

Shielding Fails!

Continued discussion of subspace physics.

Subspace is an odd set of the laws in our dimension, specifically it is missing the electromagnetic force, or half the electroweak interaction. This means that things that rely on that interaction in our dimension fail spectacularly in subspace.

Shielding against this affect is the most important part of a functioning subspace disc after the transitions into and out of subspace. The disc is a carefully arranged array of projectors that are encased in a titanium-doped mono-ceramic, capable of withstanding rough use. Only voltage difference need be applied over the arrays, meaning that only two wires enter and deliver the current to the array. Of course the shield produced by the disc isn't generally referred to specifically, as shield generally refers to electromagnetic deflector arrays mounted on some ships.

When shielding fails, the matter inside the previously shielded space loses all electromagnetic attraction. It completely ionizes every atom, stripping every single electron to travel tangentially from its current orbit. Depending on the entrance velocity and the distance traveled, small scale experiments, of course over very large distances, showed that electrons sometimes collided with nuclei and stuck due to the strong interaction but more often did not, preserving the shape over shorter timescale tests. When exiting subspace at the limit it was determined that there were two different reactions from electron-connected and non-connected nuclei.

The electron connected nucleus shot away at high velocities relative to the distribution of the protons and overall mass. The second more common reaction was stripping any intercepted matter of electrons, often very few in the vacuum of space causing the remaining ionized atoms to disperse in a cloud, and releasing large amounts of energetic electromagnetic radiation as electrons fell into orbits, bumping into other nuclei and bumping them further. This causes a majestic and most-likely deadly "space aurora."

This can actually be used to advantage, currently large facilities have been replacing their aging fusion reactors with Inverse Subspace Generators (ISGs). Careful shaping of the subspace arrays can produce a bubble of subspace.

Two atoms are shot into the bubble at a single point so that the nuclei collide and stick together. The resulting nucleus exits the bubble and tries to grab electrons from a carefully maintained cloud of gas, as well as possibly shedding a neutron. The resulting cascade results in a bright flash that is captured by the electro-photonic (different than photo-voltaic hence the name difference) material on the chamber walls.

The theory as to why this works in producing energy is that it is actually stealing energy from subspace. This is not yet confirmed, and the consequences haven't been mapped out, however, most people think this resource is inexhaustible at the rate any foreseeable future holds. And this might be balanced out by the amount of photons that "disappear" into subspace as ships produce photons in large quantities.

The reason this hasn't been implemented on ships yet is because of the gravitronics involved. For the atoms to enter and exit the bubble in the same room the process requires an extremely flat gravity gradient, meaning that the system has to measure and adjust to any gravimetric anomalies as often as atoms are sent into the bubble, sometimes exceeding several GHz. Maintaining this system can sometimes take up a significant portion of the produced power.

Another reason ships can't use this, as long as there aren't any break-throughs, if they go into subspace themselves the two arrays tend to cancel each other out, causing lopsided and ineffective shielding on experiments.

Some large experimental ships have added an ISG that is turned off before entering subspace and then turned on again after exiting. This is more of a supplemental generator as the main power needs are met by a fusion reactor that is not limited by the same constraints.

Lastly, I would quite enjoy a discussion on these fictitious ideas.

Monday, October 27, 2014

Sub-space Equations: Drift and Pointing

This post is continued from the previous one, talking more about depth and a bit about velocities.

Generally a well pointed ship will come out tangential to the orbit of a gravity well, and fall into the correct orbit with only minor modifications. Of course if it comes in perpendicular it could cause the ship to quickly fall to its destruction, or be at an escape velocity on the other side of the gravity-well pointed away from the mass.

But there are a few problems with traveling in subspace, other than getting your heading just right. There is drift. Drift causes a ship to essentially lose depth in subspace, meaning that potential energy in our universe is lost.
Where po is the starting depth and 256 is a conversion variable related to a light year and deltaD is the distance traveled in light years. This equation hits about 0 depth in 32 light years, this causes ships to pop out of subspace with 0 potential energy, essentially an energy vacuum. If not deeply inside of a gravity well this can cause a catastrophic energy re-balance that can overwhelm a ship's shields.

The dimensionality of this equation might seem suspect. Shouldn't entering into subspace deeper in a gravity well mean that the entity is deeper in subspace if it is so tied to our gravity wells? This equation is more of an approximation. It is much more of a logarithmic curve, however, in order to go "deeper" it would probably be inside the event horizon of a small black-hole, which brings up an interesting point. If a ship were able to survive the tidal forces could it escape a black-hole via sub-space?

The other end is much more enigmatic, what happens if the ship dodges the near certainty of exiting subspace? It might continue to build up further and further debt, but would the probability of exiting subspace further approach 1? Or would it hit a peak and then taper off causing a very small percentage of ships to be trapped until reaction mass was exhausted and the shielding failed?

The largest problem for day-to-day travel is getting orbits correct because velocity is conserved throughout the process, and you would rather be in the same orbital position relative to a gravity well as the one you left, meaning that your potential energy in orbit would be conserved, except for drift.

To get an idea of the distances and tiny angles involved, and the real limiting factor to navigation apparent: precision.

Where e_r is the acceptable error in radians, d_a is the acceptable error in distance from target, and delta D is the distance to travel. Delta D and d_a are the same units so let's take travelling 1 light-year.

Going through this we find that it is about 1.5x10^-5 radians, somewhat simple for computers to handle. In fact being off by the whole distance of the Earth to the Sun is definitely not something you want to risk, at least not toward the sun. The real limiting factor is the control dynamics of the craft itself. Most small craft will have large reaction wheels, able to spin a craft with high accuracy and maintain pointing during minor shifts of mass. Milliseconds before entry into subspace a smaller more accurate set can get the error down to around 10^-10 or 10^-11. On the other end this translates to 1500 km. Still not great for a tight orbit on a planet.

The smallest consistent error was set by a crew on a small commercial packet ship averaging 350 km off of set targets. The captain of the ship went on to found his own company for delivering extremely sensitive information very quickly, because normal space still takes time to maneuver if you aren't in the right orbits even with powerful engines. He did very well for himself up until a small adjustment made while maintaining the reaction wheels deviated his craft by 1000 km causing a spectacular splash in the target planet's ocean, and disintegrating the craft. The resulting tsunami was quite small as most of the energy dissipated in different frequencies.

Larger ships use maneuvering thrusters to get a general alignment and then use several large sets of reaction wheels to precisely align, though generally their accuracy is 10^-9 or slightly better meaning they need to give themselves more than 150,000 km error.

I mentioned shield failure, I will discuss consequences next time.

Friday, October 24, 2014

Sub-space Equations: Velocities and Depth

As promised from the previous post: Equations.

First let's start with an equation describing the relative velocities between normal space and subspace.

Where K is the constant of about 8.24x10^23 kg^-1m^2. v_n is the velocity relative to all influential gravitational bodies, depending on the precision needed. The more precise this needs to be the more bodies need to be considered, such as for sensitive instruments going into subspace to observe anomalies. If it is an emergency or the computer power is limited, which in this time period would probably mean there was also other emergencies, less n bodies can be used. Of course if the right bodies aren't considered it can be catastrophic in when a wrong exit point is determined.

The second equation calculates the "depth" that the system will enter into subspace, or the potential energy.

Where G is the gravitational constant, m is the masses of objects and d is the distance from those masses. With further posts I will give further equations on depth, but essentially this is the potential energy of the system and can be modified for the benefit or detriment of ships in subspace.

I think this is all about geeking out over playing with numbers. And setting up equations which I can throw real numbers into to see what happens.

Thursday, October 16, 2014

Fun Physics and Fiction

Time is the enemy. Dramatic events are unfolding, but will our hero be able to save the world, yet again, in time? Used to dramatic purpose time is a very useful tool as there are very few people who haven't been up against a deadline, real or constructed. They can feel the tension building, the expectation of succeeding slowly winning out over defeat or receding as panic sets in. Split seconds described in paragraphs that take ten times as long to read than to happen in book time.

Time can also be used dramatically in a longer term view in science fiction. Where will we be in so many years? Time travel is another way, and then there are the books that stitch together star systems with relativistic travel, decades long travel between stars and everybody that you knew when you headed out is long dead and hundreds of years have passed. To me it is depressing, or rather actually scary. When we travel on Earth we get those same affects, but since we are no where close to the speed of light we don't notice that those people sitting still are relatively older than us.

So most science fiction that isn't diligently exploring what life would be like without a go-faster-than-light card creates a way to travel faster than light without the time dilation problem. However, they try to limit themselves with problems such as gravity wells and where and how the drive works. So let me introduce an idea for FTL drive that is limited in its own way, I might have equations...

First, the Basics

Discovery of a universe "below" ours that our universe is wrapped around. It is smaller in the fact that things can move much faster through it. The main feature of the neighbor universe, sub-space, is that it has no electromagnetic force, but it does show the effects of our universe in terms of gravity. A ship going in unprotected would just fall apart as all the electromagnetic forces holding atoms and molecules together would collapse.

Also gravity wells are a major player, essentially entering and leaving the sub-verse must be done at a similar gravitational potential energy. Coming in at higher potential energies cause major power stripping coming in, lower potential energies means the possibility of overwhelming systems with incoming energy. There is no limit to how far inside or outside a gravity well a ship must be however, orbital and absolute velocity relative to mass and proximity affect the velocity in the sub-verse.

What do I want my timelines to be in the universe? Do I want to make it such that we are retelling old sailing tails and battles on the high-ocean? Or skipping across the universe seemingly unfettered? I think I have enough restrictions to play with both logically and dramatically, but I also want a bit of down-time, unable-to-do-anything time that my characters might need to experience. I think reasonable speeds would be about two light-years in and hour, meaning about 8760 times the speed of light. Of course that is highly variable due to the gravity well that you are exiting.

One of the major problems with early travel using this drive was pointing and timing. Hitting a star's or planet's same orbital potential in such a way that you weren't just plummeting toward its surface but instead tangent and able to establish an orbit was extremely hard, however the first wave of extra-solar colonists risked it.

I will continue these thoughts in the next installment.

Monday, October 13, 2014

Jobs, Again and More

Probably the hardest thing to do from Hilo: track down a job on the mainland, and get hired.

So if you know anyone with a software position open and willing to hire me, then let me know. In the meantime let me talk about something else. I believe earlier in this blog I have mentioned how much I used to dislike writing for a class, but how much I enjoy writing outside of class. Well it's sort of like data and my post about subways really culminated a few of those into one. I definitely didn't have much trouble with the writing, just some of the code kept me boggled for longer than it should have.

I really do like writing, must have gotten it from my mother: Bird Banter, and her companion gardening column with a few less years: Garden Gossip. I really like quite a bit of science fiction as well. I really enjoy hard science fiction where there might be far out concepts but the author integrates them into the story, possibly using a new concept to turn a bit of the plot around. So maybe as I am trying to find a job that might have sounded outrageous 50 years ago I should aspire to finish writing a story. Yes, finish a book in fact, as I have around eleven thousand words and that is mostly from writing a few hundred at a time during lunch breaks.

It isn't about the words though... It's about the story and the concepts, where one explores the other. Anyway, I suppose there is enough out there about literature and what it is attempting to do. But I would like to write a story.

Had a really good conversation with a hiring manager today, after the previous paragraphs were written. I think either writing this or having the "long-term career goals" question asked must have jogged something loose. I responded that I want to be a science fiction writer, of course being a software engineering manager he might have thought that was crazy, except when I went on to describe what I meant:

I would love to be a best-selling sci-fi author, writing about fantastic technologies and using my stories and characters to explore them, or vice-versa. But what are many of these technologies going to need in the short term? Software, plenty of software, even if you call it firmware, more on that in a minute. Whether or not I ever become a successful author I still want to positively influence the world with my software contributions, essentially an author of progress.

Let's say that I have to fall back on the idea of being an author, where the heck am I going to start? Well first off, I won't give up software right away. There are still projects to contribute to, things to improve. What I really need is motivation in both arenas, I need to find people/projects that are responsive and have a clear idea of what's happening next. I need to remember what I am doing, very good blog post by The Art of Manliness blog on remembering.

I definitely need to remember my path, but I need to remember the important structures beneath it, it must be supported somehow. And I just need to remember good responses to screening questions. I felt a bit underwhelmed by a second screening call today from another company. I completely forgot my long-term response from earlier, "Why do you want to work for us?" Uhhh... because you sound like a great company. Great, Bryan,  no vision, no passion, not going to get a job that way.

Writing, how should I go about it? I follow a couple of sci-fi writers, some good ideas there definitely. And I have been voraciously reading many different levels of books, from self-published essentially, to well edited best-sellers. Well I am going to need practice, and one way to practice is to write more! I really need to find a group that I can bounce ideas off of, but who knows when I will find that.

Now back to writing changes in the world. Or that's what I will keep telling myself.

Saturday, October 11, 2014


So we really haven't been fencing much. I think I will have to read my own posts on fencing before getting back into it. We have a really odd schedule right now that is coming to an end come the end of this month. In the meantime we have picked up archery.

Isn't that just another expensive sport? Well, sort of.

We started getting interested as I usually get interested in sports. One of our friends was trying it out and I went to the indoor range to find out what the atmosphere about it was, and if it was any good. I've done archery in scouts, but hadn't really thought about it except in passing. So I went.

Managed not to kill anyone despite the fact that I got an arrow to bounce back to within a few feet of the line. But it was a 70 lb compound bow, not my cup of tea. So I went and hung out more to watch and learn, but my friend stopped coming. That's how it's happened for all but table tennis: Someone introduces me to a sport and then immediately gives it up.

I bought a bow made in South Korea, for probably too much for a starting bow, recurve, 40 lb draw, and just over 3 foot in length. I bought some arrows and tried it out. I was terrible, couldn't hit the 4x4 foot bale at 20 yards consistently. Jessie tried a ~20 lb bow from one of the regulars and liked it so much we bought it off of him for a good price.

That was about a year ago, since then we have enjoyed the company of other archers and forgotten quite a bit about fencing. Recently we have been consistent and accurate enough to get pretty good groupings, it feels really pretty good to get about 10 arrows all within a few inches of one another without the massive amount of equipment. My arms and back are much stronger and can deal with about an hour and a half of shooting. All in all, a good time.

Now that we are getting back some of our time in the future it will be fun to get back to fencing, but we are definitely keeping archery as one of our sports.

Tuesday, October 07, 2014

Progress Stunted

As a modern man, and a person living in our world I understand the need for tolerance, but of what should we be tolerant? And where should we draw the line?

Tolerance is along the same lines as endurance. There are many things which we must endure day-to-day, year-to-year, and through the generations.

When someone needs to live somewhere for a job or just because they can't get away (or wants to live there, but that is outside of this consideration) they must tolerate the weather. Snow, ice, cold, rain, heat, drought and many other conditions that we must live with. What do people do with the weather?

First they tolerate it and buy appropriate clothing, coats, hats, shorts, whatever would make walking through the weather more tolerable. This is a terribly important step that many animals don't have the luxury of, in fact it is why humans can survive in such varied conditions, even those who are acclimatized to Hawaii can make do with a few extra layers farther north.

We insulate ourselves and take other precautions in all ranges of weather. Sure there are those perfect days where clothing can be minimal, but for those of my translucent-skinned brethren we still apply a layer of sunblock. We could go to an extreme insulating ourselves by building shelters that we never have to leave, but there are so many problems that arise. Though those the problems have been mitigated via "happy-lights" and carefully choosing the winter crew for Antarctica, we are never truly insulated from our surroundings. Astronauts on spacewalk have air leaking out of joints in the gloves still, and space is far more hostile than any but the most severe weather event on earth.

Second they talk about it. Complain in fact. But what is the most boring conversation starter? "How about the weather?" Or something similar. When you run out of stuff you are comfortable talking about you fall back to the weather. Now that gets me thinking about how often I talk about the weather, maybe because I don't know people well enough to ask them about other things? Anyway, it is a mundane subject that is generally considered inoffensive because everyone is in the same boat. When farmers are hit by the drought, they talk about it. The news loves talking about extreme weather events...

Third they try to change it. Cloud seeders, I haven't seen any hard evidence that these are successful all the time, in fact in the past there may have been many frauds just trying to get farmer's dwindling money. The funny thing is that we are changing the climate every day. If in chaos theory it merely takes a butterfly flapping its wings half-way around the world to start a hurricane, than how much more air is each one of us moving by driving, or even shaking hands on a business deal?

So where am I going with this? Tolerance.

We can never shut out the weather completely, so we must tolerate it, and everyone has their own coping mechanisms from embracing to denying. But should we tolerate other things that make up our climate? And by climate I mean social climate. I am sure there are tens of thousands of blogs that call for tolerance for one thing or another, and many others that are specifically or generally intolerant.

It is also quite over-discussed in groups of geeky friends, philosophy classrooms, and governments (at least I hope). Tolerance is a buzzword in religion too, and should we be tolerant of other religions? Well let apply weather as an extended metaphor to social climate:

Let us say you have a social climate that you are most comfortable with, friends, family, and people you deal with often enough to know well and not feel anxious around. You also have a meteorological climate in which you feel most comfortable. Stick with me because we are going to dump both, you move to a new town with much different weather, and know absolutely no one. As a bonus this town is far enough away that all the things you might consider social norms are turned around and nothing seems to make sense.

I speak with some experience on this point, I moved from Wyoming to Hawaii, quite a contrast. So what do you do? Insulation, talk, and try to change it? Maybe, let's just say I have decent insulation, but not so much I don't know what's going on and can't talk to locals, however my closest friends tend to be more like me than like locals. I talk about the funny things I hear, with friends, but also with locals, we share a common experience. Have I tried to change it? Sure, I have helped introduce the sport of fencing to quite a few people on the island. So yes I have adapted to the climate as well as having a bit of impact on it.

Now here is a bigger leap, a deeper dig: Tolerance: the ability or willingness to tolerate something, in particular the existence of opinions or behavior that one does not necessarily agree with. So can I be a tolerant Christian, and moreover a tolerant Christian man? I just lost a whole bunch of nonexistent readers right there.

I believe the answer is a resounding "YES." There, hopefully some people who were about to leave stayed around. And now I am about to lose more: Christ said in reference to the Romans occupying Judea, that Christians must give up their cloaks, carry items, and turn the other cheek. Severe paraphrasing, either ask me or go look it up and correct me. In that age this was tolerating the Roman military who had the right to ask citizens for materials and labor, and could hand out minor corporal punishment on a whim.

So several thousand years later we have gone from being an oppressed minority to pressing our views on others. Personally I don't believe this is oppressing, not that we are killing about it much anymore, unlike some other religious groups I could name. However, we as the "Church" (big 'c' to indicate all Christians) are insistent on pressing our views on people via government laws and group social pressure. "Be like us and enjoy the fruits of our hypocritical actions." I am sure that someone is thinking that right now without the "hypocritical." But Christianity is not about the human institutions, despite nearly two millennia of human institutions "devoted" to Christ.

Tolerance isn't about institutions unless it comes to protecting someone from intolerance. There are many movements that I disagree with in today's social climate, and no, not all of them are considered socially liberal. I could list them and tell you why I disagree with them, but I do not write to change your mind about any particular movement, I write to change or reinforce your attitude about tolerance.

In fact, I urge you to tolerate any non-threatening movement. Now the other half of my non-existent audience are leaving. Tolerance is not agreeing, it isn't acquiescing. I have my morals and believe that some things are wrong and that other even-more-wrong-headed individuals will push things down slopes without friction.

But that is the thing, tolerating, not acquiescing. I am not about to go out in a Wyoming winter storm without proper gear, and I certainly won't acquiesce to the universe's insistence on normalizing temperature gradients to match the bitter cold of my surroundings, forget it. That's what irks me is the insistence of acquiescence from any side. "You must tolerate us!" But what those groups do, including my groups, is change the tone to imply that tolerance is no longer enough, tolerance now means to acquiesce to demands. And the real victories come when they are able to institutionalize it so that any stray word or thought can be persecuted.

It is based on individuals, not institutions. I know many people with different views. Do I tolerate their views? Yes, except where they are in danger of harming themselves or others, and generally a gentle explanation can straighten out obviously ridiculous views, unless they are in the fanatic camp. Should I insist they follow my views? Should they insist I follow their views? No, in most cases. But then how can I be a good Christian and convert those people I come in contact with? Isn't that the whole idea? Yes it is, but it isn't for me to push my views on you, I may express them and step out of participating in something that I have opposing views to. I believe that if you find my views to be better than yours for some reason that you will come talk to me, and I wish others would extend the same courtesy to others outside their own groups.

We should discuss, we should debate, we should listen, we should tolerate, but we should also be aware that tolerance is not acquiescence. I reserve the right to my views. In a previous post I recounted how my views on data changed, there are many other things I have negative views of that could in fact change, and I don't believe any of them are harmful to any individual, but I must strike down this pedestal before it gets too high.

So now the reason behind this epic: I believe the protesters blocking TMT groundbreaking are incorrect and too late. That is my opinion, I ask you tolerate it, but in this I would also love to go beyond just insulation and talk about it, and maybe even change the climate. I will leave it at that.

*updated November 2016 with mostly links

Saturday, October 04, 2014

To Take or Not to Take the Subway

Introduction to Data Science Capstone
The New York Subway system spans 842 miles of track with 468 stations and transports about 5 million people a day. But those are just large numbers, first we need to get a feel for the data. What does the precipitation look like throughout the year? What does the ridership look like on an average day? Can we predict ridership with the given information?

The dataset I am using to answer these questions is turnstile data from 2013 collected over the entire city, however, I am limiting mine to the most consistent station for reporting data: Wall Street. I also signed up for an API on Weather Underground to get hourly data for the same year. Let’s take a look at precipitation throughout 2013.
Precipitation in New York 2013: Shows rain and snow amounts throughout the year.

This graph has all 365 days of the year with the precipitation of each hour arrayed along the horizontal axis. The alpha of the dots shows us the intensity of the rain in the hour. The most intense rain in an hour recorded 1.06 inches. Given that information and that the alpha setting needs to be between 0 and 1, I normalized all the rest of the rain data against that value giving us an alpha gradient and a decent visualization.

It gives us a sense of the precipitation, but we also need a good view of ridership throughout the year. However just creating a line graph or a scatter plot won’t really help us visualize the data. The first thing to try would be averaging all the hours to get an average day’s ridership.

After some file size issues, data had glitches from when a turnstile would cut-out, and I also found that the reporting time of every four hours to be problematic, especially when a single turnstile would report during an off hour.

I found that the only station to get nearly continuous updates was the Wall Street station. It is only missing a total of 21 reports for the entire year. So I ran it through a series of filters designed to get rid of steps and glitches and came up with a graph that looks much more like a counting line:
Running it through some SQL queries in pandas I got the average ridership per hour so that we can compare it to our rain data, at least visually.

The interesting thing with the entries data is that morning rush hour is quite a bit more than the really early morning, but it never really slacks off at lunchtime, in fact there are more and more entries until a peak of about 2,750 entries at 18:00.

But does this correspond at all to our weather data? As it has one similar axis I could flip it so that the axes match up and then have a dual axis along the bottom.

This is the result with the red line as the average day/workday throughout the year and the orange line denoting the weekend days. It might just be a fluke of the climate, but it seems June’s heavy rain-showers correspond quite nicely with the uptick in traffic at 18:00. However, this is also about the time most people are leaving the non-residential district to head back home after a day of, ironically, pushing numbers to get better results.

Although visually biasing the graph really doesn't tell us what we want to know, and this is where we turn to regression.

It seems that the best predictor of subway foot traffic is time. With over a thousand more hours predicted within 1000 commuters compared to using just weather, which is more than ten percent of the data set, deciding when to ride the subway will affect the number of people riding with you more than if it is raining.

However, I was wondering what a different algorithm might be able to tell us. What might a machine learning model be able to predict, possibly trained on different parts of the year? So I turned to scikit-learn to go through a few different algorithms.

At first I tried a Bayesian Ridge which is very similar to the algorithm used in the class and that I modified to use on this data. Its results were nearly the same, although I was able to get some very interesting overfitting using the entirety of the year as a learning set. Otherwise it seemed that somewhere in April had a set of data that was the best at predicting the entire year. When I checked the r-squared value I was a bit surprised: 0.23. Not as good as I was hoping despite the graphs showing much of the predictions within one thousand passengers.
AdaBoost compares quite favorably to the regression used in class. It also has an r-squared value of nearly 0.53.

Both algorithms have problems predicting the evening rush-hour numbers, however AdaBoost seems to recover for later in the night.
I also tried the AdaBoost algorithm which sets up multiple prediction agents rather than just weight the inputs. With this algorithm the predictions got an r-squared of 0.52, quite a bit better. Still the driving factor behind all of these predictions is time, and I could probably get an even better result by separating the weekends and holidays out of the weekdays. And a problem with the AdaBoost algorithm: it isn’t completely repeatable with minimum controls as it gives a different distribution each time, but seems to maintain the r-squared value.
Difference of ridership against prediction (red) and precipitation (blue) show that not the prediction algorithm was not greatly affected by precipitation, getting predictions nearly right and quite wrong. The deepest red dots are predictions off by nearly twice the highest average traffic, about 6000 riders.

Generally it seems that the majority of people using the Wall Street station use it all of the time, that and the people who don’t either continue walking or using other transportation despite most weather.

The last thing to consider is the different mapreduce options. First of all the whole dataset isn’t truly that large, at 60MB it certainly bogged down my computer and ran into system timeouts, but removing those limits and putting in a few more Gigs of RAM to deal with copies easily would go a long way to solve these problems, not really in the realm of true need of MapReduce.

To get into this realm we might look at something that could be a bit more complicated. Real time tracking of entrances and exits on the subway system to get a much finer grain picture of the entire system would certainly get close to qualifying, especially if you wanted to more than just swim in the data.

Another future application of real time systems maybe routing traffic with a pay-or-get-paid system for taking city-wide data and destinations and crunching it down to route traffic. If a person needs to be somewhere quickly they can pay a fee to take a faster route while those taking slower routes would get paid to wait. With the advent of self-driving cars this might not be as onerous as one might think as the people stuck in the slow traffic could possibly work, that or their company can pay for them to use the faster routes to be in more quickly.

But dealing with hundreds of thousands of inputs every minute would definitely need a system that could split up the tasks between servers or possibly server farms. Being able to predict problems or heavy loads becomes a necessary issue.

Most of these traffic issues, whether subway or cars, can be predicted most easily using times and events, the only time that weather is truly going to affect traffic patterns will be in extreme or catastrophic situations. If you are pinning your hopes on getting a seat on the subway because of weather patterns then you might get a seat just before the subway shuts down during a snow day.

References: - Udacity course - python plotting library - Weather data - New York subway data - Regression algorithms

Featured Post


John studied himself in the mirror as best he could through tears. Red, puffy eyes stared back at him, a running nose already leaked just a ...