RSS Feeds
Posts
Comments
Photos
Twitter
technmsg

Out of the box, Apache Accumulo attempts to replicate its table metadata five times.

There’s nothing wrong with that practice, ensuring that the table metadata will be more resilient to hardware failure. Unless you have less than five DataNodes — not uncommon in the case of a small demo or test cluster — in which case HDFS will start complaining about under-replicated blocks.

Since an HDFS client can specify an arbitrary replication factor per-file, changing the default replication factor (dfs.replication in hdfs-site.xml) won’t make the errors go away. In this case, Accumulo is the client. So, assuming you’ve got three DataNodes, login to the Accumulo shell as root and run the following command:

config -t !METADATA -s table.file.replication=3

Of course, if you’re setting up a test/demo cluster from scratch it’s helpful to do this sooner rather than later.

2013, A Tasting Pictoral

A brief visual trip from my 2013 tasting diary, in chronological order.

No apologies if you head to the kitchen right after…

Lunch from Pho 75
Pho from Pho 75
Reston, VA

Untitled
Grilled Thick Sourdough Breakfast Sandwich at Lowell’s (better picture)
Seattle, WA

Infused Cocktails at Tini Bigs
Infused Martinis (pre-poutine) at Tini Bigs
Seattle, WA

The Boat from The Weiner's Circle
The Boat from The Weiner’s Circle
Herndon, VA

Vader's Vegetables
Vader’s Vegetables at Cloudera
San Francisco, CA

Untitled
Assorted Steamed Gua Bao from The Chairman Truck
Palo Alto, CA

Macaroni & Cheese at Deschutes
Macaroni & Cheese w/ Chorizo at Deschutes Pub House
Portland, OR

Reggie Deluxe at Pine State Biscuit
Reggie Deluxe at Pine State Biscuit
Portland, OR

Engagement Munchies
Late-night Engagement Munchies on the Strip
Las Vegas, NV

Candy Bar
Candy Bar at 1840′s Room
Baltimore, MD

Mmm, Crepes
Crepes at La Crêperie
Key West, FL

The Obligatory Hot Dog
The Obligatory Chicago-Style Dog at O’Hare
Chicago, IL

$5 Breakfast in Virginia
$5 Breakfast at Deli Mall Cafe
Springfield, VA

Fresh House Made Mozzarella, Tomatoes, Balsamic
Fresh House Made Mozzarella, Tomatoes, and Balsamic at Open Kitchen
Falls Church, VA

Untitled
Italian Stromboli & MegaFries at Pat’s
Newark, DE

Pastrami from Katz's Deli (NYC)
Pastrami on Rye at Katz’s Deli
New York, NY

Dominican Stewed Beef and Chicken
Dominican Stewed Beef & Chicken at D’Mangu
Rochester, NY

IMG_0046
Carved Thanksgiving Turkey at Home
Maryland

Cloudera: Twelve Months In

After more than twelve months with Cloudera, I still have difficulty explaining to others what I do for a living. Titles mean different things to different people at different companies in different industries.

Perhaps some exposition is in order.

As a part of our Field Services Team, the bulk of my time is spent embedded with customers, guiding their exploration of Hadoop.

Sometimes that means installing and configuring a cluster from scratch, or perhaps stepping the customer through the process step-by-step, side-by-side. Sometimes I certify existing clusters, offering configuration and tuning recommendations to get more out of the system. Sometimes it’s being given a set of data and trying to figure out how it fits into the Hadoop ecosystem, if at all.

The Hadoop ecosystem is vast and growing every day, with many interconnected hardware and software components, each of which have a multitude of possible configurations. And let’s be honest, sometimes things just don’t work quite right the first time, even given a solid set of instructions (e.g. Kerberos, encrypted shuffle). It’s a learning process for everyone: often times the customer is new to Hadoop and the customer’s environment is usually new to me.

We work through it together.

I’m still learning, and that’s okay.

I don’t think my customers expect me to know everything there is about Hadoop, but more to be a guide through this “big data” thing. Consider, a mountaineer might not know every rock or landmark on the trek, but they’ve seen and experienced enough to have a good sense of how to proceed given the lay of the land in front of them; with every ascent and descent they add a little more breadth to their knowledge, and if needed they can call upon their team if they need additional expertise in a specific area. Such is the same of every customer engagement I’ve been on.

I get to contribute, beyond my job description.

I’ve seen the smiles of conference attendees when they get handed a Data is the New Bacon t-shirt or witness look of disbelief when I show them that yes, we do have a tofu-based variation for the pork-averse.

I’ve helped develop and bring in new customers inside and outside of the DC Beltway, gone on the road with my local sales rep, wine and dined prospects, and celebrated wins alongside the sales team. I’ve seen a few west coast ocean sunsets in the process, never a bad thing.

I’ve met countless new faces at community events, seminars, and meetups. With a little help from my amazing coworkers, I overcame a mild case of stage fright and did my first presentation at a conference.

I’ve interviewed dozens of candidates on the phone and in person, determining whether I think they’d be a good fit for the team. I’ve referred a couple people who I thought would make solid additions to the team; importantly, I feel my input matters when it comes to staffing.

I’ve liaised between customers and our support teams, and while my engagements with customers result in far more tickets being assigned to engineering, I’ve even gotten to contribute a little code and documentation myself.

Some of these things may seem trivial. Some many seem silly. But I believe that all of these things matter, and to be honest I enjoy the variety.

And then there’s a bit of fun.

Visualized, this field guy’s 2013 looked a little bit like this.

No lies, most days don’t feel like work. It’s been a good time.

I get to wear many hats.

As I’ve told my boss on numerous occasions, I don’t care what my title is so long as it doesn’t limit my compensation or mobility. And so far it hasn’t.

What does that make me? Happy.

After enabling Kerberos on a working Hadoop cluster running CDH 3u6, we ran into two problems. Tiny problems, but problems that took a long time to find and prevented the cluster from working.

In troubleshooting, we found the existing corpus of documentation and mailing list archives lacking, so hopefully the following might assist others in the future.

To be clear, we followed the CDH3 Security Guide to the letter. I had used the guide on previous installations and felt pretty confidant in it’s content.

So, here’s what we saw.

The NameNode wouldn’t start.

The relevant excerpt from the NameNode’s log:

Login failure for hdfs/hostname@REALM  from keytab /etc/hadoop/hdfs.keytab
javax.security.auth.login.LoginException: Unable to obtain password from user.

We verified settings, permissions, even went through regenerating all of our Kerberos principals and keytabs. Manual kinit worked fine, so we felt pretty good about our principals, keytabs, and the KDC. We turned on Kerberos debugging (adding -Dsun.security.krb5.debug=true to hadoop-env.sh) which netted us a little more in the log, but not much.

Config name: /etc/krb5.conf
Config name: /etc/krb5.conf
>>> KdcAccessibility: reset
>>> KdcAccessibility: reset
>>> KeyTabInputStream, readName(): REALM
>>> KeyTabInputStream, readName(): hdfs
>>> KeyTabInputStream, readName(): jobtracker.hostname.tld
>>> KeyTab: load() entry length: 88; type: 18
>>> KeyTabInputStream, readName(): REALM
>>> KeyTabInputStream, readName(): hdfs

We were stumped.

Turns out we missed a trailing space in our realm. Did you catch it in the log file above? We certainly didn’t, amidst all the verbose output that Hadoop provides, but a closer look at hdfs-site.xml revealed the following.

<property>
  <name>dfs.namenode.kerberos.principal</name>
  <value>hdfs/_HOST@REALM </value>
</property>

One extra space. So stupid. After removing it, the NameNode started without complaint. We were able to do a simple hadoop fs -put (after kinit-ing) to verify HDFS operation.

Which brought us to the next problem…

MapReduce jobs were failing.

Even submitting the most basic pi job resulted in a failure.

java.IO.IOException: Job initialization failed (255) with output: Reading task controller config from /etc/hadoop/taskcontroller.cfg
INFO error setting supplementary groups for user hdfs: Operation not permitted

We checked permissions of our MapReduce log directories, local directories, HDFS directories, group memberships, no dice.

The short version, make sure the volume containing task-controller isn’t mounted nosuid.

As it turns out, all of our non-root volumes were mounted nosuid per local security policy. In addition, our Hadoop components were living in a separate /opt volume.

Under typical Hadoop operation, this didn’t matter since all jobs were run as the mapred user, but once Hadoop security was enabled each job was to be run as the submitting user (by way of a setuid binary named task-controller). Since task-controller wasn’t being allowed to perform a setuid operation to run our task as our user, the tasks were failing.

After we re-mounted the /opt volume without nosuid and restarted the TaskTrackers, our jobs were up and running in no time!

Two months ago I started at Cloudera, a mid-sized Silicon Valley startup that focuses on open-source software.

The decision to leave my old job was hard.

To start, I had many of the same reservations that I did back in 2008 when I left a previous project for “something new, something promising, something challenging,” except this time it was harder. In the past four years I’d become an intricate part of a small team and we were responsible for some amazingly cool, important, and far-reaching things. There was no question whether what we were working on was important, and I felt that I was a valued member of the team. I had become a proponent of change, advocating for improvements to software and policy, and a go-to guy for a couple different subject areas. I felt important, like what I did mattered, and that I mattered. That’s a powerful, alluring feeling.

And yet, it wasn’t enough.

The atmosphere within SAIC changed considerably when we went public in 2006. More managerial time and effort were spent on boosting quarterly numbers to please market analysts than on customer satisfaction. With each passing CEO reorganizing the company to suit his pleasure, the company became increasingly fractured, the spirit of employee ownership and pride dissipating rapidly. I can’t even count the number of reorganizations due to change in management personnel or business priorities and for most of my time post-IPO I couldn’t have told you what business unit I was part of, less so the continually changing faces that were in charge of them.

What I do know is that the small Annapolis division that I had come to know and love was consumed by a larger organization, and this in part was a precursor to my eventual departure. The new management style was much more opaque, layered, rigidly structured, and much more concerned with profit margins than with the group of close-knit people they have just inherited from Annapolis.

The nature of contracting helped.

Being embedded in a customer’s space full time makes compartmentalizing easier. While there are plenty of split-brain contractual hoops to jump through, I effectively became an employee of whatever customer I was working for. This suited me fine, since I was always more interested in the work than who was signing the payroll checks. The customer’s goals became mine, as did their problems, policies, work ethics, organizational politics, successes and failures; when you’re in the trenches together it isn’t uncommon to have great working relationships with your customers even when things are a touch rocky with your employer.

For the most part SAIC management was content to let me do my job without any interference. Even when they didn’t, I always found my immediate supervisors incredibly supportive throughout my time with the company, in many cases sitting and working side by side on a daily basis.

Simply put, 2011 was a mess.

Tumultuous, defined. The first half of the year I was running three days a week before work, in and out of a whirlwind long-distance relationship, working on some high-profile code for a customer, staying in touch with my family, doubling up on online courses working towards a Master’s degree, keeping up some semblance of a social life, all while attempting to get a reasonable amount of sleep each night necessary to fuel my hectic schedule.

It was at this point that SAIC management decided to get involved with certain aspects of my work. I don’t want to relive the details, suffice to say mistakes were made all around and the matter wasn’t handled well. The end result was that I was left in a position where I was contractually obligated to continue to work for people with whom: I wasn’t comfortable, I didn’t trust to keep my best interests in mind, I didn’t feel understood me or my objectives.

Even after the direct involvement passed, there were still lingering reminders of how it had gone wrong and little recourse to restore things to the way they were before.

It was inevitable.

The signs were there, although I didn’t recognize them for what they were.

When you’ve got trust issues at work, those emotions don’t tend to stay at work; even if you bottle them up (while on the clock, in the name of professionalism), they’ve got to find an outlet. My personal life outside of work was affected; I completely lost my desire and motivation to write and share, which I rationalized away as due to lack of time. Something I genuinely take joy in, rationalized away for months.

It wasn’t a big secret to those close to me that things weren’t going so well. There was more tension, although I wasn’t able to put my finger on the source. I thought it was a temporary feeling, brought on by taking on too much at once (a few tough classes, particularly), and that once things settled down everything would be fine. I got through the classes, but I still wasn’t comfortable.

A leap of faith.

After more than nine years with SAIC, I found myself running out of reasons to stay.

Throughout the year I’d been hanging out with a friend who had recently accepted a job with Cloudera. He seemed to be enjoying the work and he suggested that I might as well, and that they could use someone with my skill set. I didn’t know exactly what it would be like at Cloudera or whether I’d like the atmosphere, whether the company would be around in 2-3 years, or whether I’d miss the kind of work I’d been doing, but I figured it wouldn’t kill me to learn more.

When I talked to my family about the prospect, my sister put it bluntly: “you haven’t been happy [at SAIC] for a long time.” I was taken aback, but it was exactly what I needed to hear. After a lengthy lunch meeting with my friend (and now coworker) about what I could expect, I applied for a position. In the subsequent weeks I spoke to several engineers at length, managers, even the CEO. Everyone was really enthusiastic and I felt like it could be a good fit.

I submitted my letter of resignation to my division management at SAIC — six months short of my 10-year anniversary — and instantly knew that I had made the right choice. I felt free. In the two weeks following I spent a lot of time briefing my teammates on my work and documenting the same, backing things up, preparing for my departure. There was no time for guilt about leaving, less than a month after my first serious talk about Cloudera I found myself sitting in our Palo Alto office getting the rundown on healthcare, company culture, meeting the staff, and starting anew.

Bring it on.

Last Flight of Discovery

It’s hard to express why I wanted to see Discovery make her final voyage, but I just had this feeling that I had to. A fascination and curiosity to see her in flight, without turning on a television or loading photos in my web browser.

Father/Daughter Discovery

I’ve wanted to see a night launch for years, experience night becoming day. I’m told that you could read a book for ten minutes after, so much raw energy being expended from the solid-fuel rockets. There is something intense about breaking free of Earth’s gravity and how much fiery power is required to do so, and I can only imagine that forces of that magnitude can only be fully appreciated in person.

My desire to see Discovery be ferried over Washington, D.C. is more of an emotional draw, the source of which I can’t explain. But the decision was simple, last Tuesday I took the day off and boarded the Metro, destined for the National Mall and eventually the Washington Monument.

I wasn’t alone.

One man — wearing a NASA windbreaker — told me that it was a special day for him, having been to Discovery’s first launch and every one after. One amateur photographer — lugging around a rolling suitcase full of clothes and equipment — had driven down from Michigan the night before, just for the day, before planning to drive back that night. The press was on the scene as were dozens of families, milling around the base of the Washington Monument waiting to catch a glimpse.

Until someone shouted, “there she is, 2 o’clock!”

Sure enough, we could see the landing lights of the Shuttle Carrier Aircraft (SCA) shining in the distance, growing closer by the second. There was a tension in the air, then the cheers broke out. Even with nearby airports, it was surreal to see a 747 at that altitude, following the Potomac River but appearing to be aimed straight at the Washington Monument, much less with a space shuttle bolted onto the top. As she banked to the right and flew over the Lincoln Memorial we got our first view of Discovery’s profile and all we could hear was the crowd’s excitement and the rapid-fire clicking of cameras all around.

IMG_5962

She was huge, making the even more enormous 747 (for which I have a sense of scale) look like a toy and the T-38 chase plane look like a bug. In most of the photos I took the T-38 Talon is reduced to a handful of pixels, despite being 46 feet long. Even at an altitude of 1,500 feet I could see that Discovery’s skin was far from pristine, her tiles grayish instead of black, the result of many missions into space. I’ve read that the Smithsonian intends to keep her exterior as-is, scuffs and bruises intact. This pleases me.

She banked to the left as she flew over the Marine Corps War Memorial, taking an amazing 45 minutes to make several large counter-clockwise passes around the National Mall, over us, the Capital, and the Department of Commerce.

In what I normally find to be a beautiful but self-centered city, I found dozens of strangers talking to each other about their memories of the Shuttle program, smiles abound. Small kids were perched atop their parent’s shoulders. Adults were madly dashing around the Washington Monument to follow the aircraft, just like small children would. It’s amazing to see an event bring people together from all walks of life.

I didn’t have the greatest camera gear along, my lenses are a little short to get anything super-detailed, so I opted for composition and creativity. CNET picked up one of them all the same, which is a pretty cool thing by itself. I shot a few hundred pictures, many of which I plan to turn into a flip book; the cloud cover might have been a downside for some, but for animation purposes it will help create the sense of motion.

IMG_6082

Are there better shots that captured the event? Sure. But that’s not as important to me. I was there. I felt the cool spring breeze on my skin. I heard the high pitched whine of the engines as the shuttle arced across the sky, framed between green grass, cloudy blue skies, and national monuments in between. I felt the excitement of the crowd, heard their cheers, and saw their smiles.

I’ll never forget it.

BBQ Beef at A&W

Hello, I’m a BBQ Beef sandwich from A&W Pit Beef at Duvall’s Marketplace. I am generous portion of thin-sliced pit beef atop a tasty roll, drizzled with a sweet tomato-based hickory BBQ sauce.

BBQ Beef Sandwich from A&W

I can be yours for $6 (with a bag of chips), to be enjoyed at one of many outdoor picnic tables. Pleased to meet you.

A&W Pit Beef at Duvall's Marketplace on Urbanspoon

The Panasonic KX-T3968 is a very old 900 MHz cordless phone, but still functional. Manuals are not available on Panasonic’s operating manual site, but it’s easy to program auto-dial or speed-dial numbers using either the base station or the cordless handset.

Note: Buttons to press are indicated in bold type.

To store a number on either the base station or the cordless handset, press the Program button, dial the digits you want to store, then press Auto button followed by the digit of the location want to store the number in. You’ll hear a one second confirmation beep.

To dial a stored number, press Auto followed by the digit.

For example, to store 202-555-1234 to Auto 1, you’d press the following buttons:

Program, 2, 0, 2, 5, 5, 5, 1, 2, 3, 4, Auto, 1

You can also insert pauses in the number by using the Pause button on the cordless handset or the Redial/Pause button on the base station. Just in the event you need to dial 9 and then wait, I suppose.

Both the base station and the cordless handset provide storage for 10 phone numbers, one per digit. The stored numbers do not appear to be shared, so effectively you could have 20 unique numbers. The procedure for storing numbers is the same.

Textbooks Can Be Wrong

I was doing assigned reading from Guide to Computer Forensics and Investigations (Fourth Edition). The text on page 104 concerned itself with acquiring compressed disk images as evidence, particularly the importable of ensuring that the copied data is the same as the original.

Popular archiving tools, such as PKZip, WinZip, and WinRAR, use an algorithm referred to as lossless compression.

According to the book, lossless compression is used for forensics acquisition. Lossy compression isn’t, because it alters the original data.

That makes sense.

But then came the following text:

An easy way to test lossless compression is to perform an MD5 or SHA-1 hash on a file before and after it’s compressed. If the compression is done correctly, both versions have the same hash value. If the hashes don’t match, that means something corrupted the compressed file, such as a hardware or software error.

When I read this, I stopped. I didn’t believe it. While hash collisions are certainly possible from a mathematical perspective, they aren’t likely during everyday use. But I had to see, just for my own sanity.

$ echo "Hello world." > test.txt
$ md5sum test.txt
fa093de5fc603823f08524f9801f0546 test.txt
$ sha1sum test.txt
4177876fcf6806ef65c4c1a1abf464087bfbf337 test.txt
$
$ zip test.zip test.txt
adding: test.txt (stored 0%)
$ md5sum test.zip
76e13462f780d302e0eb0246c6e4d6d4 test.zip
$ sha1sum test.zip
edfc4a3ee2082fef21dae7028d11d031eef65242 test.zip

Just as I suspected, neither method generates the same hash. I wouldn’t expect it to, even if the file were just stored within the archive there’s still content metadata that is added to the archive.

$ rm test.txt
$ unzip test.zip
Archive: test.zip
extracting: test.txt
$ md5sum test.txt
fa093de5fc603823f08524f9801f0546 test.txt
$ sha1sum test.txt
4177876fcf6806ef65c4c1a1abf464087bfbf337 test.txt

As expected, the decompressed file has the same hash as the original.

I suspect this was merely an editing error. Perhaps a non-technical editor simplified the paragraph and none of the proofreaders caught it. I hope that the authors don’t really believe the cited sentences above. I hope that digital forensic evidence isn’t being tossed out due to compressed file hashes not matching up to their source hashes.

I think the text intended to say that the hash of the original file should remain the same as a decompressed file, presuming that a lossless algorithm was used. But that’s definitely not how it sounded, and could be critically misleading to someone without previous knowledge.

The Teppanyaki Grill reminds me of Las Vegas.

As soon as you walk in, the tile floor gives way to a marble-encased wall of waterfalls. A room full of food sits behind them. You pay for the buffet and any drinks up front, then you’re led to your seat. There’s ample room for large parties, set back away from the buffet lines.

And then you go to town.

Typical Chinese buffets around Maryland might have one or two stations with two dozen dishes, but Teppanyaki goes overboard and has several hundred dishes. In addition to the usual Chinese staples, they’ve got a variety of Japanese and American dishes too. It’s typical buffet-quality as you would expect (some things are a little better than others), but there’s not much that isn’t offered here.

Various rice and noodle staples, check. Typical chicken and beef Americanized Chinese staples, check. Dumplings and egg rolls and spring rolls, all there. Pizza slices and seafood pasta and macaroni and cheese, check. Spiral cut ham. Salmon fillet. Chicken wings, plus a bacon-wrapped chicken dish that’s been cooked in butter. Crabs, clams, oysters, and shrimp, check, as well as some other shellfish that I didn’t recognize.

It continues.

They’ve got a crew of two making sushi rolls non-stop. I wouldn’t expect miracles from that department compared to the numerous options in Columbia and Annapolis, but it’s there. They’ve also got a passable hibachi grill station, with chicken, beef, shrimp, and all the usual vegetables you can load up on your plate. You don’t have all the sauce selections that you might at a Mongolian joint, but the grill man manages 8-10 plates at a time without mixing anything up.

Don’t forget dessert.

There is a fresh fruit and desert bar, complete with Oreo cookie ensemble, cheesecake, and red velvet cakes, as well a variety of pastries, brownies, pies, and cookies; don’t forget your pudding and tapioca. A large chocolate fountain sits to the side, pre-stocked with marshmallows and pineapple (although someone had the ingenious idea of dipping the aforementioned bacon/chicken contraption), located right next to the ice cream cart.

It was pretty crowded near the end of the lunch rush, but there was never an enormous line for anything. It’s organized, the mark of a well-run chain/franchise operation. The staff does a good job of monitoring the pans and replacing them when they get low. Table service was excellent, plates cleared and drinks refilled without having to ask.

Lunch runs $7 per person, Monday-Saturday 11am-3pm; $3.35 for kids 3-6, $5 for kids 7-10, free for kids under 2. Dinner prices are similarly tiered, running $11, $5, and $7 respectively. Military members get a 10% discount from the start. The local papers are publishing all sorts of $1-2 coupons, making it even cheaper to try out.

There’s something for everyone at Teppanyaki, and the price can’t be beat considering the diversity of food being offered.

Teppanyaki Grill & Supreme Buffet on Urbanspoon

Afghan Cuisine at Faryab

Saturday night we went to Faryab in Bethesda. Located amidst many excellent sounding/smelling dining options on Cordell Avenue, Faryab dishes up humble yet rich Afghan dishes without fanfare or frolic. The restaurant sits in a quiet, unassuming store front adjacent to a cigar shop. In the warmer months they have an outdoor patio. There is a nearby parking garage, free on most evenings and weekends.

I sampled quite a few appetizers and dishes from around the table:

Bulanee ($7) are turnovers stuffed with spiced scallions and herbs, and one of two vegetarian options on the appetizer menu. After a few nibbles, found these to be very light and refreshing and with a kick of pepper.

Badenjan Goushti ($21) is seasoned lamb, braised, topped with eggplant, onions, and tomato, baked, served with basmati rice. The lamb itself was very tender, falling apart in your mouth as if it were chipped beef. Not a very gamey cut of lamb; lamb lovers probably won’t care for it; lamb haters probably won’t mind it. I expected a little more flavor from the sauce, the tomato stood out among all the other ingredients.

Sabsi ($7) is fresh spinach cooked with onions and garlic. Shredded super-fine but not at all watery, my side order was sweet to the taste as if they’d used sugar-injected sweet onions. By comparison the spinach served with the lamb had a different flavor entirely; both were good, but I preferred the sweet.

Shalgham Goushti ($21) is seasoned lamb, braised, topped with tender, sweet and spicy turnips, served with basmati rice. I only got a little taste of the Shalgham dish, since it wasn’t mine; my intention was to experience a turnip for the first time, so I can’t speak to the lamb, but I was amazed by the sauce. More on this shortly…

If you take away nothing else from my experience, read the next paragraph carefully.

If you like spiced dishes, you must go to Faryab and order something with Shalgham. The sweet and spicy sauce used was probably one of the best spiced flavors I’ve ever tasted. Ever. It’s instantly sweet to the taste, and reminded me slightly of cinnamon. But then a unknown blend of spice migrates to the back of your tongue and explodes. Like a pepper might, except that it wasn’t particularly peppery. It packs a punch that will catch you off-guard, but it’s not a sweat-inducing heat. Less of a heat than a shock to the taste buds. I’ve never tasted anything like it, nor could I readily identical any of the flavors nor any of the ingredients used. That’s rare for me. Whatever is it in, it’s finely ground and not visible to the naked eye. And it’s delicious.

Enough about that, now. Overall, the fare was pretty simple and I can’t say there was a bad dish among those I tried.

There are some weird price points, where I can’t reconcile whether the appetizers were priced high or the entrees were priced low. Consider Mantu; perfectly steamed dumplings filled with a rich mixture of ground beef and onions, topped with a slightly tangy yogurt and a hearty meat sauce. If you order the appetizer ($7) you receive four dumplings. If you order the entree ($19) you receive eight, which means you’re paying $5 for an iceberg lettuce salad and one or two piece of bread.

I thought that portions were a tad small, given the cost. When in Bethesda, however…

It wouldn’t be exactly fair to compare the flavors of Faryab to that of Helmand (in Baltimore), since both feature cuisine from their respective provinces of origin (both of which they are named after). But there are some differences worth noting. The bread at Faryab was thick and heavy, very unlike the naan-like pita at Helmand. The sauces at Faryab were more tomato and vegetable based, where Helmand seems to be more broth-based. For what you pay at Faryab, you’d expect a little more ambiance; both have tablecloths and while linens, but Helmand feels a little more upscale.

Faryab on Urbanspoon

The lunch specials at China Dragon are enormous and inexpensive.

Szechuan Chicken from China Dragon

There’s better take-out to be had and the spiciness just isn’t there, but sometimes you just want a whole lot of cheap take-out (for comparison, note the size of the fork). For $5.25 you’ll get a massive dinner-sized portion of your entree, fried rice, and an decent egg roll.

China Dragon is located in a small plaza off Route 1, just north of Guilford. Purely take-out, there’s no room to sit. There’s an abundance of prepaid phone cards hanging behind the counter as well as a handful of burner phones. Also, the staff isn’t known for their friendly telephone demeanor… but they’ve got some mean lunch specials.

China Dragon on Urbanspoon

I got a direct message on Twitter last night, asking me if I wanted to go to the Hippodrome to see Wishful Drinking. I had never heard of it, but the name sounded interesting enough. I did a quick search and found that it concerned the life experiences of one Carrie Fisher, both written and performed by Carrie Fisher.

Wishful Drinking posterI was sold.

I’m not even a huge Star Wars person. I enjoyed the series for the most part, but I remember her more for playing Marie in When Harry Met Sally. She’s been in Hollywood her whole life, a fact that you’ll learn all about during the performance. But fear not, Star Wars fans, there’s plenty of stuff for you here too.

I love dry wit.

After reading the notes in the program, I knew I was going to enjoy this one; in the thank-you portion of the program, Ms. Fisher recognizes her mother, daughter, all 12 of her shrinks, and her father:

Thanks for bringing your drug dealer to my opening in Berkeley. His notes were inspirational.

Truth can be stranger than fiction.

She starts by talking about how a man died in her home. In her bed, actually. As she lounges around a cozily outfitted stage in bare feet smoking an e-cigarette, she touches on her life experiences with perfect pacing: famous parents, Star Wars, bipolar disorder, good memories and divorces, Paul Simon, addiction and rehab, her gay husband, getting old.

It’s all in there.

Both personal and comic at the same time, a wonderful mix of both personal sweets and bitters. There’s something energizing about someone speaking to a crowd about her demons and asking them for their opinion. It’s personal. It’s comic. And it’s over too soon.

The last two shows in Baltimore are today at 1pm and 6pm. I suppose you could catch it on Netflix or Amazon if you don’t want to drive to Ohio, but it wouldn’t be the same. You better hurry.

I know next to nothing about Korean food, except that I really enjoyed a recent lunch at Shin Chon Garden. While I didn’t elect to cook my own meal at the table, it was still a new experience.

Banchan at Shin Chon

Before your order is taken, banchan are set out. These are small appetizer-sized dishes meant to be shared by the table. We noted pickled cucumbers, mushrooms, sprout salad, kimchi, and a few other zingy dishes that we couldn’t readily identify. No matter, we left most of the dishes empty by the end of the meal.

Beef Bulgogi Box at Shin Chon

I ordered the Beef Bulgogi Box ($8), expecting a smaller portion. I was wrong. Horribly wrong, but in a wonderful way. In addition to the entree, the box contained two pork dumplings, a spring salad with citrus dressing, two hand rolls, sticky rice topped with sesame seeds, some sort of pickled vegetable (beets, we thought), and a vinegar/soy dipping sauce. I passed the rolls off to a coworker who would better appreciate them, but otherwise I indulged and enjoyed everything there. The beef bulgogi itself was tender and flavorful, served mixed with julienne onions, sesame seeds, and green onions. All was marinated in a rich and savory broth; I made a point to finish it off as a marinade for the rice I had left over.

There’s a lot more to explore here.

Located in Ellicott City off of Baltimore National Pike, the first thing I noticed when we arrived was the stack of Korean newspapers and journals stacked on the floor by the door. As far as authenticity goes, that’s never a bad sign. Neither is the lack of English-speaking patrons and staff; if you don’t speak fluent Korean you’ll be in the minority. The menu is written in both English and Korean, but without much in the way of description. Our server managed the basics of our order without too much difficulty, but our follow-up questions took a little more effort. We were fortunate to have a native-speaking coworker with us, who translated, described what each dish would be like, and made recommendations for us.

One tip: since barbeque is done out in the eating area, expect pungent aromas and strong flavors. If you have clothes that you don’t want to absorb the smells of your meal, you shouldn’t wear them to Shin Chon.

If you insist on English-speaking servers or knowing exactly what you’re eating down to the individual ingredients, Shin Chon probably isn’t for you. I wouldn’t let that stop you, however, since you’ll be missing out.

Shin Chon Garden on Urbanspoon

Chris’ Restaurant is a small, family-owned business, offering a mix of Latin, Mexican, and Salvadorian style food in an informal sit-down setting. Both traditional and Spanish breakfasts are offered all day. You get a lot of flavorful food for your money, here. Put simply, it’s one of my new favorites.

I was hesitant at first, mostly because the location has a history of failed restaurants (pizza, Mexican, etc) dating back to the late 90′s. But after a hearty recommendation from Damien, I figured I’ve give it a try. Ever since my personal ban of Rivera’s (which was tough, given their tasty cuisine), I had been without a go-to Mexican restaurant in the Severna Park area. Not anymore.

Chicken Burrito at Chris' Restaurant

Whenever trying a new restaurant, I start with something familiar and simple. I want to see how the kitchen is going to take a few ingredients and put their unique spin on them. Nothing is worse than waiting for a complicated dish only to find that you don’t care for it at all, so I opted for a chicken burrito.

But I’m getting ahead of myself.

Complimentary Chips/Salsa at Chris' Restaurant

It isn’t a proper experience unless complimentary chips and salsa are provided. I don’t care about authenticity here, we’re accustomed to munching while we peruse the menu. Given that most Mexican menus have more or less the same set of staples, we take longer to look at it than we would other types of restaurants, probably because we’re searching to see if there’s anything new or unique offered. The irony here is that even if you see something unusual after scouring each item’s description, you’re likely to order your standby favorite.

A basket of warm chips were provided, along with cups of curtido and salsa. Thin and crunchy chips, unsalted. The salsa was thin like you might find at Chevys, but far more smoky and spicy.

As I waited for food to be prepared, a nearby couple sipped their horchata and conversed with the server in Spanish.

I ordered a Pork Pupusa ($1.75) to get started, a traditional Salvadoran dish quite suitable for snacking. Effectively a tortilla pancake stuffed with your choice of filling and seared on the grill, the meat was well flavored without overpowering the tortilla (which is more than just a delivery device, here). It was served with a cup of curtido and an orange-colored sweet tomato sauce, both tasty.

The Guts of a Chicken Burrito at Chris' Restaurant

The Chicken Burrito ($7.99) was enormous, packed with lots of shredded chicken along with rice and beans. The chicken was moist, rice soft, beans mellow. You’ll definitely need a fork and knife to handle this, especially considering that the whole thing was covered in a light flavorful mole and shredded cheese. Lettuce, tomato, guacamole, and sour cream were served on the side.

Interior of Chris' Restaurant

The exterior of Chris’ isn’t much to look at, nor does the front signage give much away. Set in a small block of businesses along the side of Ritchie Highway (adjacent to the Donut Shack), there’s a walk-up counter with a view into the kitchen. A small sign advises customers that they can bring in their own beer/wine, so long as the 2/person limit is observed. A dozen or so two- and four-tops are nestled back along the depth of the building. Cultural artifacts and flags adorn the walls and a small television is broadcasting news in Spanish.

As usual, delicious meals are freely found right around the corner without frills or fanfare.

Chris' Restaurant on Urbanspoon

Older Posts »