Sans Hack Challenge 2016 – the 7th audio

Right, so let’s continue last post and look at getting the final audio file.

Running nmap -sC analytics.northpolewonderland.com gets me something very interesting, namely a .git directory, that’s accessible over http:


443/tcp open https
| http-git:
| 104.198.252.157:443/.git/
| Git repository found!

I can get its contents like this:
wget --no-check-certificate --mirror -I .git https://104.198.252.157/.git/
At this point I only have the git history, but no files. No problem, I can get the files as follows:
git checkout -- . restores all files from git history

Continue reading

SANS Hack Challenge 2016

Ok, time for something new now. For the past 2 years I’ve been doing (rather irregularly) security assessments. It’s quite a new thing to me, compared to the 14 years I’ve spent doing software engineering. Still I’ve already learnt a lot and got some great results and huge customer satisfaction in my security related assignments.

This year, just as last year, I took part in the SANS holiday hack challenge: https://holidayhackchallenge.com/2016/ . The SANS team spent a lot of effort creating a whole browser game, combining graphics, music, gameplay and hacking. It turned out great!
Since the challenge is over, I can now publish my writeup.
The plot of the challenge is that someone kidnapped Santa, so you have to find Santa and the villain who did it. As a starting point you get a Twitter and an Instagram handle: @SantaWClaus
Eventually you will need to find 7 mp3 audio files, which are stored on various hosts, and complete the challenges from the browser game.

Continue reading

Time and Material Projects

Last time I’ve written about fixed price projects, and was quite negative regarding them. In this article I’ll look at another type – the time and material project. Here the client is invoiced regularly for the hours and expenses for the development.

You’ll spot a problem with it right away – the provider will have an incentive to make the project as long and as costly as possible, as that guarantees more revenue.
Another negative aspect for the customer – you keep paying your invoices every month, but that doesn’t mean you also have a software you can use, nor that you ever will. At some point you may decide the project is no longer feasible, but the money is already spent. So all you’re left with is a big hole in your budget. At least with a fixed price project, the provider has the contractual obligation to deliver something, at a defined deadline.

As to the advantages for the client, you have more flexibility with regards to the scope, as you are not tied to a rigid contract. You can change your mind, re-prioritize the features and add or drop features without all the hassle of a fixed scope.

Of the two described methods, as the provider, I prefer time and material. It is crucial though, to have a good, trust-based relationship with the customer. How do you achieve that? Through previous projects, but that’s a chicken and egg problem. Communication and transparency are vital here, and using an iterative/agile project methodology helps a lot in that regard.

The client needs to be able to see what has been done, what is being done, and how much is there left to do. What are the risks, what are the challenges, what will we implement in the next sprint/iteration? The customers needs to be a central part of that. You think a task will go over budget? Communicate that as early as possible, don’t just hope it’ll even out. All this goes towards building a trust relationship with your customer. Which means your project will have a much greater chance of succeeding.

Me and OSGi

OSGi is much like a religion. I’ve even come up with the ten commandments of OSGi:
1. the OSGi way is the only way
2. all 3rd party code that you touch must be OSGi compliant
3. everything must be an OSGi bundle
4. every bundle must have a public part (api)
5. every bundle must keep its private parts covered
6. thou may not rely on the start order of your bundles
7. thou may not rely on a service being present
8. other bundles may be loaded and unloaded at any time
9. thou shall use OSGi in every project
10. thou shall bring the ways of OSGi upon the non-believers

Okay, it’s a bit of a stretch, I admit it. What I’m saying though about it being like a religion, is that it demands of you to follow certain rules. But if you look around you, very few people actually follow them. And if you try to follow them, you’ll find it very difficult.

Why is that? You can do your project the OSGi way, but you will use 3rd party code. Which may or may not be OSGi compliant. If it’s not, you’ll have to bundle it yourself. You can use bnd to do so automatically, but that doesn’t always work, especially for larger projects with lots of dependencies. So sooner or later you’ll find yourself spending 1-2 days trying to coerce some library into OSGi form. With maven it would be a 30 second job, but you’re doing OSGi so you’re stuck with that.

That begs the question: why do OSGi? It was initially conceived for embedded systems, which needed 100% uptime and the possibility of updating modules independently of one another, without stopping the system. Is your project about that? Then you have a valid case for OSGi. You aren’t likely to run into the problems I mentioned, since you’ll likely restrict your use of 3rd party code for reasons of size and performance.

However, most of us work on plain old Java applications. Is modularity good? Well yes! Do you need to update modules without restarting the application? No, you don’t. Do you replace service implementations at runtime? No, you don’t. Do you load and unload modules dynamically at runtime? No, you don’t.
That means for most projects the only gain is modularity, for which you pay a hefty price in the complexity of managing dependencies. There are simpler ways to ensure proper modularization.

Some web applications will need to update without downtime, but for those OSGi is not the answer either. If you need high availability, you’ll already have a cluster or a bunch of servers behind a load-balancer, which you can update without taking the application down.

Conclusion: OSGi is good for what it was originally intended for, but a real pain for most other projects.

Free vs commercial Tools

I’ve seen a few software companies where it’s important that the tools they use are free, or at least as cheap as possible. By tools I mean IDE, File Manager, Document management, Issue Tracker and so on.

These are tools that you, as a developer, use every day, 8 hours a day, 5 days a week.

When it comes to this kind of tools, I want to have the best there is. Most of the times, getting the best will cost money (not always though, sometimes free tools are better than commercial ones). However, even tiny improvements in efficiency will pay off the cost of the tool very quickly, given that you use it so often. Or oppositely, small annoyances from bad tools add up very quickly, and you become frustrated and unproductive. Instead of solving the problem at hand, you’re fighting the tools you must use. That’s a lose-lose situation.

Your company may say – hey, the tool you want costs $1000 per year, that’s a lot of money. For 100 developers, thats $100k per year! But let’s look at it another way. What does a day’s work of a developer cost? Let’s take $1000 a day. That means even a 5% improvement in efficiency will pay itself off in a month! A 1% improvement will make up for the investment in 5 months, so it’s still very much worth it. As a bonus you get happy developers, which means productive developers.

So, do the math, and get the best tools out there for your developers, no matter if they are free or not!

Fixed price software projects

At my workplace we have quite a few fixed-price projects. Projects involving the government, local administration or official institutions are just about always like this.

Now, to be clear from the start, I’m not a big fan of this kind of projects. Why? Because I think they are fundamentally flawed. They have you (as the provider) and the client pulling in opposite directions from the very beginning.

It already starts when you submit your offer. You have to offer a competitive price. So you’ll offer the very minimum that covers the requirements of the client. If you don’t, and try to think of a fancy solution, you’ll be too expensive, and won’t win the project.

Once the project starts, it’s just a matter of time until the first “In Scope or Change Request” discussion with the customer. For them, everything is part of the agreed scope, so they don’t need to pay. You’ll need to push in the opposite direction. You can’t just play nice and do every change they request for free, cause you’ll lose money.

Sometimes you’ll have a great idea on how to implement a feature in a nicer/more maintainable/more user-friendly way, but it’ll cost more. How do you bring this idea to the customer, especially if they are very strict on sticking to the budget, or are very tough negotiators when it comes to project scope? You risk another of the discussions I’ve mentioned above.

That’s not all. End-to-end, same project, same scope will be more expensive with a fixed price model. The risk is higher for the provider, so you’ll calculate a higher margin, to make up for that risk. Of course there is also the risk that you grossly underestimated, and will have to pay out of your pocket to get the project done.

So why are they used so much? Well, it’s only natural to want to know what something will cost you, before you buy it. If you go to the shops, you’ll look at the price before putting the item in your shopping cart. If you want to build a house, you’ll want to know if you can afford it. Same goes for software projects. But there’s a catch. Software projects are very dynamic. You may agree on a scope and a price, and 1 year later, when the project is finished, end up with something that doesn’t match your needs any more. So you’ll have kept the project within budget, but ended up with something you can’t use.

In the end, what a fixed price guarantees, is that the customer will spend the agreed price, and get some software in return. Which may or may not match what they actually wanted.

Unfortunately the rules in the public sector won’t change soon, so these projects are a kind of  necessary evil, that we’ll keep encountering for a long time.

Running a headless raspbmc media server on Raspberry PI

I’ve been banging my head with this for a few days now and finally got it to work, so I thought I’d share, as documentation about it is quite sparse.

The idea was to use my Raspberry PI as a headless media server. Headless as I don’t have a TV, so I’d use it mainly for listening to music.

Step 1: Install raspbmc on my Raspberry. That went smoothly.
Step 2: Connect the external HDs where my music is stored to the Raspberry PI. These get mounted automatically under /media/ with raspbmc.
Step 3: Edit ~/.xbmc/userdata/sources.xml to add my music library:


<sources>
    <programs>
       <default pathversion="1"></default>
    </programs>
    <video>
        <default pathversion="1"></default>
    </video>
    <music>
        <default pathversion="1"></default>
        <source>
            <name>Musica</name>
            <path pathversion="1">/media//MYBOOK/Music</path>
        </source> 
    </music
    <pictures>
        <default pathversion="1"></default>
    </pictures>
    <files>
         <default pathversion="1"></default>
    </files>
</sources>

All looks fine and dandy, do a service xbmc restart and … NOTHING! All the googling did not help much either, as the documentation only describes how to add a source from the XBMC UI, which I didn’t have any access to.

A lot of swearing occured…

I then found this thread on the xbmc forums: http://forum.xbmc.org/showthread.php?tid=132919&pid=1263543#pid1263543. Ok, so it seems I can edit the database files directly!! Gooooood, I like that! Let’s go:


sqlite MyMusic32.db
insert into path('idPath', 'strPath', 'strHash') values (1, '/media/MYBOOK/Music', null);

So far so good, now how do I get the damn thing to scan for media???

JSON-RPC Api to the rescue!

A quick http://192.168.1.9:8080/jsonrpc?request=%7B%22jsonrpc%22%3A%20%222.0%22%2C%20%22method%22%3A%20%22AudioLibrary.Scan%22%2C%20%22id%22%3A%20%221%22%7D later (for those without an inbuilt URL Decoder {"jsonrpc": "2.0", "method": "AudioLibrary.Scan", "id": "1"}) , and my Raspberry is happily scanning my music files! Yeehaw, I deserve a beer for that!

Organising unit-test data

How do you organise the data for your unit tests? In most non-trivial applications you will have a fairly complex domain model. Again and again you will find yourself needing to create instances of such complex objects for your tests.
Over the time I’ve used 3 methods for this:

  1. Copy-Paste. It’s quick, and being unit-tests it’s not such a no-go. It also keeps tests independent of one another (as opposed to method 2). The downside is when you need to update the test data, for example if you add a new field – you’ll need to go over all the copies and update them.
  2. TestDataHelper. Have a class with static methods that create and pre-fill your objects with data. For any updates you only need to change one place. The downside is that it couples your tests between them. If you need different data for one test, and you change your Helper class, it may break other tests.
  3. TestBuilders. This is my favourite method right now. You create builder classes for your complex objects, with parametrised methods for child objects. This way you have one central place to make changes, but your tests are independent, as they don’t share data.
    Here’s an example:

    Catalog catalog = new CatalogBuilder()
    	.addProduct("Chair", "Wood")
    	.withCategory("Furniture")
    	.withCategory("Kitchen")
    	.addProduct("Sofa", "Leather")
    	.withCategory("Furniture")
    	.withCategory("Living Room")
    	.build();
    

Which method do you use?

Windows RTF Editor with Swing

Ever tried using an RTF editor component for Java/Swing? I did, and I was never satisfied with the ones I found. Neither were the users. One of the reasons was that pretty much all of the users were working on Windows and had become accustomed to the native RTF Editor component from the application we were replacing.
So I thought, why not use the Windows component from the Swing application? How hard can it be?
As a note, I’ve only tried this in a private test project, what you’re about to see is no production code.

The Java side is quite succint:

public class RTFEditor {

  {
    System.load("c:\\myRTFEditor\\rtf.dll");
  }

  private static native String showRTFEditor();
}

There’s just the native method that will display a modal dialog with the RTF component and will return the text that the user entered.

Getting to the native stuff (I used VC++ 10 with Visual Studio)

The header file, rtf.h, generated with javah.exe

/* DO NOT EDIT THIS FILE - it is machine generated */
#include
/* Header for class com_me_RTFEditor */

#ifndef _Included_com_me_RTFEditor
#define _Included_com_me_RTFEditor
#ifdef __cplusplus
extern "C" {
#endif

namespace rtf {

  JNIEXPORT jstring JNICALL Java_com_me_RTFEditor_showRTFEditor(JNIEnv *, jclass);
}

#ifdef __cplusplus
}
#endif
#endif

And the implementation, rtf.cpp. I got the string converstion code from Stackoverflow, unfortunately I don’t remember exactly from where.

#include "stdafx.h"
#include "rtf.h"
#include "rtfForm.h"
#include

using namespace std;
using namespace System::Text;
using namespace System::Runtime::InteropServices;

namespace rtf {

  JNIEXPORT jstring JNICALL Java_com_me_RTFEditor_showRTFEditor(JNIEnv * env, jclass)
  {
    rtfForm^ form = gcnew rtfForm();
    //show modal dialog
    form->ShowDialog();
    // Encode the rtf text from the component as UTF8
    array^ encodedBytes = Encoding::UTF8->GetBytes(form->getRtf());

    // prevent GC moving the bytes around while this variable is on the stack
    pin_ptr pinnedBytes = &encodedBytes[0];

    // convert to jstring to return it to java
    jstring jstr2 = env->NewStringUTF(reinterpret_cast<char*>(pinnedBytes));
    return jstr2;
  }
}

You surely noticed a “rtfForm” is referenced here. I used the UI Designer for it, it’s simply a form with a RTF Editor component inside. Here’s the code for it (rtfForm.h).
I won’t include the code for rtfForm.cpp, as it contains just the include for rtfForm.h

#pragma once

namespace rtf {

using namespace System;
using namespace System::ComponentModel;
using namespace System::Collections;
using namespace System::Windows::Forms;
using namespace System::Data;
using namespace System::Drawing;

///
/// Summary for rtfForm
///

public ref class rtfForm : public System::Windows::Forms::Form
{
  public:
  rtfForm(void)
  {
    InitializeComponent();
  }

  String^ getRtf()
  {
    return this->richTextBox->Text;
 }

  protected:

  ~rtfForm()
  {
    if (components)
    {
      delete components;
    }
  }
  private: System::Windows::Forms::RichTextBox^ richTextBox;
  protected:

  protected:

  private:
  ///
  /// Required designer variable.
  ///

  System::ComponentModel::Container ^components;

  #pragma region Windows Form Designer generated code
  ///
  /// Required method for Designer support - do not modify
  /// the contents of this method with the code editor.
  ///

  void InitializeComponent(void)
  {
    this->richTextBox = (gcnew System::Windows::Forms::RichTextBox());
    this->SuspendLayout();
    //
    // richTextBox
    //
    this->richTextBox->AcceptsTab = true;
    this->richTextBox->Anchor = static_cast((((System::Windows::Forms::AnchorStyles::Top | System::Windows::Forms::AnchorStyles::Bottom)
    | System::Windows::Forms::AnchorStyles::Left)
    | System::Windows::Forms::AnchorStyles::Right));
    this->richTextBox->Font = (gcnew System::Drawing::Font(L"Courier New", 8.25F, System::Drawing::FontStyle::Regular,     System::Drawing::GraphicsUnit::Point,
static_cast(0)));
    this->richTextBox->Location = System::Drawing::Point(1, 3);
    this->richTextBox->Name = L"richTextBox";
    this->richTextBox->Size = System::Drawing::Size(477, 217);
    this->richTextBox->TabIndex = 0;
    this->richTextBox->Text = L"";
    //
    // reForm
    //
    this->AutoScaleDimensions = System::Drawing::SizeF(6, 13);
    this->AutoScaleMode = System::Windows::Forms::AutoScaleMode::Font;
    this->AutoScroll = true;
    this->ClientSize = System::Drawing::Size(479, 221);
    this->Controls->Add(this->richTextBox);
    this->Name = L"reForm";
    this->Text = L"reForm";
    this->ResumeLayout(false);

    }
  #pragma endregion
  };
}

And here’s a screenshot of it working
rtf

Unused module dependencies – updated

I’ve updated the ‘Unused module dependencies’ plugin.

You can now choose to ignore modules with no java source files, so that modules that aggregate other modules are not shown.
There’s also a new tree view for the results, which is easier to interpret than the plain text output. As before, code’s on github.
Binary is here, or if your employer thinks you’re too productive having access to github and has blocked it, you can also get it here (rename to .jar after download)