Machine Learning and Data Science

We've used Apache Spark to extract insights from Kaggle's Survey "The State of Data Science & Machine Learning" where Kaggle gathered more than 16,000 responses.

For us, the main objective of this small project was to learn more about the state of Data Science, as an interdisciplinary field. That is why we've used the publicly available dataset from Kaggle. In addition we also wanted to experiment with Apache Spark thus we've used this tool to process and extract the insights presented below.

Gathered insights

1. Unfortunately the existing gender imbalance from the technology field is also present in the data science field. Only 16.9% of respondents are female which leads us to believe there is a big difference between the number of female data scientists and male data scientists.
2. Python is the most recommended language to learn first for people aspiring to become a data scientist.
3. Kaggle is one of the most popular learning platforms for data scientists.

Who are the data scientists?

When talking about gender, out of the 16,000+ persons, 82.6% are male data scientists and only 16.9% are female data scientists. And 0.5% chose not to answer the gender related question.

Most of the data scientists who answered are under 35 years old. Distribution can be seen in the below bar chart. In addition 10% of the data scientists who answered 60 years old or over.

Looking at the countries of the data scientists who responded the survey we can see that 25.1% of them are from United states, 16.2% are from India, 3.5% are from Russia and 3.2% are from United Kingdom. As seen below, the rest of 52% of the data scientists are from other countries such as Germany, France, Spain and many others.

Most of the data scientists who answered the survey are already employed (96.2%). Looking at the highest studies of the data scientists we can see that 28.7% have their Bachelor's degree, 37.5% got their Master's degree, 14% got their Doctoral degree and only 10% have no studies.

Survey Results

When data scientists were asked which technology they are more excited to learn next year the top five choices were: TensorFlow, Python, R programming language, Spark / MLlib and Hadoop/Hive/Pig.

Python is without doubt the first programming language to learn for a person aspiring to become a data scientist, it was the most recommended language to learn for a new data scientists by the respondents of the survey.

Data scientists were asked to respond from where they get the datasets they use to practice data science skills. As seen below most of them use some kind of data aggregator such as Kaggle, Socrata or data.world. Here are the results

1. Dataset aggregator/platform (i.e. Socrata/Kaggle Datasets/data.world/etc.) - 26%
2. Google search - 13%
3. University/Non-profit research group websites - 11%
4. I collect my own data (e.g. web-scraping) - 10%
5. Github - 9%
6. Government website 8%
7. None or other - 23%

Out of the 25% of the data scientists who answered the question "How long have you been learning data science?" about 85% of them are learning data science for less than 2 years.

The survey respondents were asked to choose one or more learnings platforms or ways which they use to improve their data science knowledge. The three most popular choices are: Kaggle, Online courses and YouTube Videos.

Out of the 23.6% of persons who answered the question whether Big Data knowledge is important to get a data science job 57.4% of them consider it nice to have, 37.9% of them consider it necessary and 5% consider it unnecessary.

The full list of results for the survey can be found on Kaggle's website here

References

1. Kaggle ML and Data Science Survey, 2017 A big picture view of the state of data science and machine learning.
2. Kaggle Insights - 2017 The State of Data Science & Machine Learning

Book review - Java 8 in Action

Java 8 was released in the first half of 2014, and it's awesome! The most notable Java 8 features are Lambda expressions, Stream API and Date/Time APIs. There are also a lot of other improvements like default methods which allow addition of methods to interfaces without breaking existing implementations and allow multiple inheritance of behaviour, but not state. Annotation on Java Types and repeating annotations were also introduced in Java 8.

Even if I was using some new features of this update for some time, I felt it was time to get a deeper knowledge of Java 8. There are several books already published on this topic. I've chose "Java 8 in Action: Lambdas, streams, and functional-style programming" based on the good ratings and high number of positive reviews at the time searching for a book, and let me say it, I don't regret it! The book was amazing!

Java 8 in Action is easy to read, but it keeps you excited throughout the whole book. The authors have done a really good job explaining the new features of Java 8. Throughout the book, new concepts are followed by small exercises which can help you to validate your understanding of the concept. A learning benefit for you is that each chapter ends with a good summary of what was discussed.

The book is split into 4 parts, first part is called "Fundamentals" and it contains a short introduction about why these new features are relevant to you, as a developer. In this part you'll learn about "Passing code with behavior parameterization" and get introduced to Lambda expressions. Second part is called "Functional-style data processing" in which Streams are introduced, explained and exemplified. You will be taught how to "collect data with streams" and you also get to learn a lot about parallel data processing and performance. Third part is called "Effective Java 8 programming", in this chapter you'll be introduced to Default methods, to Optional as a better alternative to null, some new concurrency utilities and the new Date and Time API. Forth part is called "Beyond Java 8" in which you learn more about Functional Programming and in which there is an interesting chapter called "Blending OOP and FP: comparing Java 8 and Scala", you'll like it!

As you'll read the book you will realize that it is not just about Java 8 features but you will learn a lot of other things as well. I strongly believe that after reading this book you will not only be able to use Java 8 features but you will also become a better programmer.

I really enjoyed reading this book and I strongly recommend it to you as well. Enjoy it!

Book review: The Pragmatic Programmer - From Journeyman to Master

These pages published in 1999 contain a tremendous amount of wisdom for all programmers who wish to become software craftsmen. The book contains a collection of ideas, observations and recommendations for programmers who aspire to raise the bar of professional software development.

While reading this book you will feel like you are having a conversion with two very skilled software craftsmen, you might agree or disagree with their ideas but you cannot argue their judgment.

The authors did a nice thing introducing each chapter with a small context attached, also when new concept or section is introduced it has references to things to which it relates. Also there are interesting exercices at the end of each section.

When reading about portability concerns you'll find links to dead technologies like CORBA or OS like Windows 2000, this will definitely make you reflect on how fast technology has evolved. You'll also appreciate that while sharing knowledge about various topics, the authors shared some good jokes as well.

The book has eight chapters, some of the most interesting topics discussed are: estimations, there are amazingly useful tips about estimating, debugging. Also things to do before a project starts, and topics about pragmatic projects and pragmatic teams. At some point in the book you'll also find explanations for Liskov Substitution and Demeter’s Law.

As a software developer you'll find that even if this book was written many years ago, the shared wisdom is still relevant today.

Microservice and Monolithic architectures exemplified

We'll use an example of an application to illustrate the differences between Microservice architecture and Monolithic architecture.  Let us take an e-commerce application which has the purpose of selling products online. In general all applications from this category require functionalities for browsing through available products, purchase them and placing orders which are later managed by administrators. And of course most of the e-commerce websites have some content which must be easily editable at any time from an administration dashboard.

Monolithic architecture

In the below diagram it is presented an e-commerce web application built following a monolithic architecture. As seen in the below diagram the data of the web application is stored in a single database. Products, website content, orders placed by customers and inventory information all stored in the same place. The web application has multiple purposes, first and most important to allow customers to browse through products and allow them to buy, secondly managing orders, content and maybe offering customers the possibility to create an account for managing different settings and subscriptions.

In this case the application built on a monolithic architecture offers both a web interface written using HTML, CSS and JavaScript and an Application Programming Interface which can be used by the clients such as Android or IOS smartphone apps.

E-commerce web application built using Monolithic architecture

Scaling for this kind of applications can be done both horizontally and vertically. The later means that the production machine will get its hardware improved by adding additional RAM memory, disk storage or adding a better CPU. Scaling vertically can be achieved by installing the same version of the application on multiple nodes and putting a load balancer between the nodes and the browser clients or API clients.

Microservice architecture

The below diagram shows a possible way to build the described e-commerce web application in a microservice architecture. As seen below the application was split into multiple small services each having its own purpose, managing orders, managing products and managing content. 

Each small service has its own database which contains the generated or used data by that particular service, so basically we do not have a big database but rather multiple small databases. This represents one of the big advantages of microservices because scaling of the database horizontally becomes possible. Of course that a data consistency issue appears when a complex flow, which passes through multiple services, fails somewhere in the middle of the processing. Assuring data consistency requires now a lot of extra effort, something which was previously easily achieved using database transactions.

The earlier mentioned API Gateway stands in front of the clients and incorporates the API offered by all microservices such that the clients know only about that API and are not aware of the other services behind it. This can be compared to the façade design pattern in which a single entry point for the clients is offered. Following this architecture makes it easier to later change the implementation or the API of the services as long as the API of the Gateway still remains the same.

Using Microservice architecture offers more options for scaling the application, for example one can install half of the services on one node and the other half on the other node. One can install each service on a separate node or even install each service twice on two different nodes, but then most probably a load balancer is needed.

E-commerce web application built using Microservice architecture

In future posts we'll have look over the advantages and disadvantages of Microservice architecture and over its use in practice.

Introduction into Microservice architecture

The purpose of this post and future related ones is to understand which are the advantages and disadvantages of Microservice architecture. This blog post is meant to get you familiar with this topic and represents a first step towards reaching the end goal.

Understanding what microservice architecture is and then identifying the advantages and the disadvantages of this architecture, it is first required to define some concepts like software architecture and software pattern. 

Software architecture refers to the high level structures of a software system, the discipline of creating such structures, and documentation of these structures. These structures are needed to reason about the software system. Each structure comprises software elements, relations among them, and properties of both elements are relations. The architecture of a software system is a metaphor, analogous to the architecture of a building. Wiki

As microservices architecture is an architectural pattern let us have a look over its definition. An architectural pattern is a general, reusable solution, to a commonly occurring problem in software architecture within a given context. Architectural patterns are similar to software design pattern but have a broader scope. The architectural patterns address various issues in software engineering, such as computer hardware performance, limitations, high availability and minimization of a business risk. Some architectural patterns have been implemented within software frameworks. Wiki

Because the microservice architecture comes in contrast with the monolithic architecture let us shortly review what a monolithic architecture is. 

In software engineering, a monolithic architecture describes an architecture in which the application single-tiered software application, its user interface and data access code are combined into a single program from a single platform. A monolithic application is self-contained, and independent from the other computing applications. The design philosophy is that the application is responsible not just for a particular task, but can perform every step needed to complete a particular function. Martin Fowler article

Today, some personal finance applications are monolithic in the sense that they help the user carry out a complete task, end to end, and are “private data silos” rather than parts of a larger system of applications that work together. Martin Fowler article

Microservice architecture can be defined as an architectural pattern or style in which a software system is developed as a group of smaller services, each running in its own process and communicating with each other using lightweight mechanism such as HTTP. The services should be built around business capabilities and should be independently deployable. The services should be small, highly decoupled and should focus on doing small tasks, facilitating a modular approach for building a system.

Motivation

The development of monolithic applications becomes slower as the application grows, so does grows the frustration of the developers. Large applications are hard to manage and most often doing a small change takes days to identify the impact and hours to write the code, afterwards it might take few days to pass review sessions and run automated suite of tests. These issues are some of the causes which made developers to welcome the microservice architecture which breaks down things into manageable pieces.

Following this architecture means that a big team can be split into smaller teams organized around microservices, these small teams become autonomous and fully responsible of their developed service. Fear of change will not prevent developers to fix issues or create new features, besides this the efficiency and the development speed is visible increased.

Developers feel more comfortable when managing smaller code bases than managing a monolithic one, this means that creativity is stimulated and development frustration is less likely to appear.

Another great thing which is achieved using this architecture is that developers are kind of forced to develop the application into a modular way; it also allows reusing services over multiple applications.

More about microservices..

In future posts we'll discuss more about Microservice architecture vs Monolithic architecture, advantages and disadvantages of Microservices and putting this architecture to practice.

StreamInsight components

Streams of data

Just as Microsoft SQL Server was designed to allow developers to manage static data, StreamInsight was designed to work with streams of data. But what does a stream of data me

an? Well a stream of data is a sequence of pieces of information, for each such piece of information a certain time is associated with it. Usually the associated time is the date-time of creation.

Such streams of data can be produced by countless devices which vary from smoke sensors, temperature sensors to smartphones, robots, web applications, hosting servers or trading applications. 

Event

An event can be defined as a basic unit of data processed by the StreamInsight server, each event encapsulates a piece of information thus we can say that a stream contains a sequence of events. Each event consists of two parts, the header and the payload.

The header defines the event kind and temporal properties of the event. All the temporal properties are application-based and supplied by the data source rather than a system time supplied by the StreamInsight server. All the timestamps use the .Net DateTimeOffset data type, also StreamInsight normalizes all times to UTC date-time automatically.

The payload is a .NET data structure which holds the data associated with the event, the fields of the data structure can be defined by the developer. Each field can have a .NET data type e.g. int, float, string etc.

Query

The same way as a fisherman uses a fishing net to catch fish from a river, we can use a StreamInsight query to retrieve relevant information from a stream of data. The results of the query are received incrementally for as long as we need.

One can define a myriad of queries starting from simple ones like selecting all events which fulfill a certain condition, to more complex like selecting certain events which appear in a window of 3 minutes.

The main difference between StreamInsight and a database is that StreamInsight never stores the data, a query is kept active all time while the server is running. Every time a new event appears it triggers a new computation and generates a new result. Of course if we are interested we can store the results of the queries.

Source

Devices which are data producers become sources of data for the StreamInsight application. One can define multiple sources of data which go into the StreamInsight server and against which queries are executed.

Sink

We have sources of data which become streams of data and queries which are executed against them, but how can we get our hands on the query results? Well we can define a sink in which StreamInsight will send the result of the defined queries. Also in this case we can define multiple custom sinks, some might represent a conventional database or an user interface in which users can acknowledge the information immediately.

StreamInsight components working together

As seen in the architecture diagram from above, one can define source of events like smartphones, fire sensors, smoke sensors, temperature sensors, server logs or event historical data. This platform allows developers to aggregate all these events by defining LINQ queries, the results of these queries are then passed to any developer defined sink like monitoring devices, monitoring applications or even data warehouses.

StreamInsight

StreamInsight is a platform developed by Microsoft which allows developers to create and deploy complex event processing (CEP) applications. This platform is based on the existing .NET Microsoft platform and it enables developers to implement robust and highly efficient CEP applications. There are a lot of possible event sources, some of the most relevant are:

• Financial trading applications
• Web analytics
• Manufacturing applications
• Server monitoring applications

One can use this platform to easily create tools to monitor data from multiple sources for meaningful patterns, trends, exceptions and opportunities. Analyzing and correlation can be done incrementally while data is produced (in real time) without storing it first, which translates in to having a low latency application. As a source of events historical data can also be used.

Key Benefits

In the following I will try to talk about the most important features and advantages offered by this platform.

Highly optimized performance and data throughput

StreamInsight supports highly parallel execution of continuous queries over high-speed data because it implements a lightweight streaming architecture. The use of in-memory cache and result computation done incrementally provide an excellent performance with high data throughout and low latency. In StreamInsight all processing is automatically triggered by incoming events based on defined queries. Also the platform provides the functionality for handling out-of-order events and in addition static reference or historical data can be accessed and included in the low-latency analysis.

.NET development environment

Microsoft created the .NET development environment in which programming languages like C#, tools like Visual Studio and services like SQL Server can be easily integrated and used for applications development while still keeping the loose coupling between them. StreamInsight is included in this environment in which one can easily develop fast and robust applications. Developers can write their CEP applications using C#, leveraging the advanced language platform LINQ (Language Integrated Query) to create queries.

Given the fact that there is a large community of the developers already familiar with these Microsoft technologies the cost and time of the development of a CEP application is significantly reduced.

Flexible deployment capability

StreamInsight platform provides two ways of deployment scenarios. First is a fully integrated into the developed application as a hosted (embedded) DLL. The second way is deploying StreamInsight as a stand-alone server with multiple applications and users sharing the server. This means that one can develop multiple, independent, applications which use the same StreamInsight instance. The CEP server runs in a wrapper such as an executable or the server could be packaged as a Windows Service.

Extensibility

StreamInsight allows developers to extend its functionality by giving them the possibility to define their own operators, functions and aggregates to be used in queries and define specific event types against which to run the defined queries.

One of the great things about StreamInsight is that it was designed to seamlessly integrate with any domain specific business logic. This means that the platform does not come with any implemented functionality for specific business sectors but it allows developers to plugin any specific business logic.

CEP Query Visualization and Analysis

Microsoft StreamInsight provides a stand-alone Event Flow Debugger which is a powerful GUI tool that enables visual inspection of a continuous query. One can use this graphical tool to quickly inspect the query tree, replay data processing and perform analysis.

Latest version

Currently the latest version of StreamInsight is 2.3, this was released together with SQL Server 2014 on the first of April 2014. Release 2.3 contains only a licensing update, so any code written against the previous version, 2.1, will still work.


In the future posts about StreamInsight I will present some of its most important components.

Robert Rusu

Introduction to event stream processing

Today most of the businesses are actively monitoring data streams and application messages in order to detect business events or situations and take time-critical actions. Even though plans are made for businesses, events are the real drivers of the enterprise today because they represent changes in the state of the business.

Unfortunately, as it happened in case of data management in pre-database days, every usage area of business events today tends to build its custom infrastructure to filter, process, aggregate and propagate events.

Building efficient, scalable systems for monitoring and processing events has been a major research interest in recent years. As new technologies rise and existing ones are expanding the sources of relevant events is growing exponentially. A lot of technologies have been proposed, including Data stream management, complex event processing and asynchronous messaging. 

One can observe that all these systems share a common processing model but differ in query language features. Besides, some applications might have different requirements related to the consistency of the data which might translate in tradeoffs between insensitivity to event arrival order and system performance. It is clear that some applications require that events are processed in the order in which they arrive or were created, while others are more concerned with high throughput. If exposed to the user and handled in the system, user can specify the consistency requirements per query and the system would adjust itself at runtime to guarantee consistency and manage system resources.

Event stream processing use case

As an example let us consider a financial services company that actively monitors financial markets, individual trader activity and monitors financial markets. Having a desktop application, a trader can track a moving average of the value of an investment portfolio. From the business perspective it is required that the average is updated continuously as stock updates arrive and trades are confirmed. A second application running on the trading floor would extract events from live news feeds and correlates these events with market indicators to infer market sentiment, impacting automated stock trading programs. 

The query would filter patterns of events, correlated across time and data values. In order to bring value to the business, this application needs to provide the information as soon as possible, late events might result in a retraction. Meanwhile a third application might be running in the compliance office monitors trader activity and customer accounts, to watch for law violations, bad intentioned actions or institution guidelines. This queries might run until the end of the trading day or even until it finished processing all the events from that day. These applications carry out similar computations but differ significantly in their workload, requirements for consistency guarantees and response time.

The example illustrates that most real-world enterprise applications are complex in functionality and might incorporate different technologies that need to be integrated and are required to achieve high accuracy and consistency. In following posts I will write about a solution which can be used for building applications from the area of event stream processing. The solution is a platform developed by Microsoft and it is named StreamInsight.

Setup CAS server tutorial

Prerequisites 

For setting up the CAS server the following must be installed:

• Java 7 - JDK
• Tomcat 7
• Maven 3

I assume that you have knowledge of all these technologies and I will not go into many details about them.

Setup CAS server project

CAS uses Maven to build a deployable package which can be installed into a Java servlet container. In this tutorial we'll use Tomcat as a container.

Also CAS server makes use of Maven overlay which makes the setup very easy and provides default configurations and flows which can be easily updated to your needs.

To make things faster I will provide below the pom.xml which contains only the dependency to the cas server webapp and the configured maven overlay build plugin. Importing the pom.xml as a maven project into an IDEs such as Intellij will generate an overlays folder which contains the defaults of CAS server, browsing through them will help you understand how it works. 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.blogspot.robertrusu.cas</groupId>
  <artifactId>cas-server</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>war</packaging>

  <name>cas-server</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 <cas.version>4.0.2</cas.version>
  </properties>

  <dependencies>
        <dependency>
            <groupId>org.jasig.cas</groupId>
            <artifactId>cas-server-webapp</artifactId>
            <version>${cas.version}</version>
            <type>war</type>
            <scope>runtime</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.6</version>
                <configuration>
                    <warName>cas</warName>
                    <overlays>
                        <overlay>
                            <groupId>org.jasig.cas</groupId>
                            <artifactId>cas-server-webapp</artifactId>
                            <excludes>
                                <exclude>WEB-INF/cas.properties</exclude>
                                <exclude>WEB-INF/classes/log4j.xml</exclude>
                            </excludes>
                        </overlay>
                    </overlays>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.2</version>
            </plugin>
        </plugins>
        <finalName>cas</finalName>
    </build>
 
</project>

Configuration of CAS server

You will need to copy into yourTomcatInstallationPath/conf/cas folder two configuration files, to be easier for you can take them from github and later change them to fit your needs.

• cas.properties contains CAS configuration which can be changed at runtime without the need to build the web application again. Of course after changing the configuration you will need to reload the application or restart tomcat.
• log4j.xml contains the logging configuration for CAS server

It is very important to change in cas.properties the configuration which points to the location of log4j.xml otherwise the server will not be deployed successfully.

log4j.config.location=yourTomcatInstallationPath/cas/cas/log4j.xml

You will need to create in your project a spring configuration xml file which will replace the one from the cas-webapp dependency at build time. The file name must be propertyFileConfigurer.xml and its location must be inside your project at:
src/main/webapp/WEB-INF/spring-configuration/propertyFileConfigurer.xml
The file contains the location of the CAS server configuration which will be picked at runtime, when deploying the webapp in Tomcat. The content is the following (just replace yourTomcatInstallationPath):

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xmlns:context="http://www.springframework.org/schema/context"

       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd

       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <context:property-placeholder location="file:yourTomcatInstallationPath/conf/cas/cas.properties"/>

</beans>

Build it and run it

• Build the webapp using: mvn clean install
• Copy the obtained cas.war file into yourTomcatInstallationPath/webapps
• Start tomcat (startup.bat)
• Visit http://localhost:8080/cas

You will see the default look of the CAS server together with a login form. To test it you can use the dummy username "casuser" and its password "Mellon".  
These are configured in next xml file overlays/org.jasig.cas.cas-server-webapp-4.0.2/WEB-INF/deployerConfigContext.xml in the primaryAuthenticationHandler bean. 

If you are not able to access the CAS server web application after you deployed the war file into tomcat checking the container logs or the CAS logs will probably help you to fix the issue. 

The end!

As you've seen I did not get into to much details and kept it short. I will try to make some posts on how to further configure the server to use some various authentication handlers or how to register cas clients. If you have any suggestions please leave a comment below. 

To better understand CAS I suggest that you also read the following article CAS - Central Authentication Service which contains an introduction into CAS and also a lot of useful resources to get you started. 

 Robert Rusu

Update: Even if at the time when I wrote this tutorial the CAS Server had version 4.0.2, the same steps can be applied, maybe small adjustments might be needed like using the latest configuration samples.

CAS - Central Authentication Service

Introduction

This post will give you a brief overview of what is CAS - Central authentication service as a protocol and CAS solution which is an solution for web services implemented by JASIG - Java in Administration Special Interest Group.

Before you dive in, there are some things you need to know like:

• What is multi-sign-on? We have multi-sign-on when we have multiple web applications, each having their own login form. Most probably for each of them you need to use different usernames and passwords
• What is single-sign-on? We can describe this as being able to login only once in order to access multiple web applications. From my point of view the most obvious example is Google, one needs to login only once to access all Google services like Gmail, Google Drive, Google+ or Google analytics.

CAS - the protocol

CAS is a single-sign-on protocol which allows users to access multiple web services by providing only once their authentication credentials, usually their username and password.

The involved entities in such a protocol would be the CAS server, the registered to CAS web service and the client web browser. Obviously the CAS server would hold an CAS application instance. A registered service is a service which will become accessible by the user after he successfully logs in, a user would use a web browser to access the service.

One of the best way to get you started with understanding CAS protocol is to follow the sequence diagram which is available on JASIG's website at http://jasig.github.io/cas/4.0.x/protocol/CAS-Protocol.html.

CAS - the application

CAS, the application, developed by JASIG is an open source software that implements CAS protocol.

The application consists of a server component which is written in Java. CAS developers also used Spring Webflow and Spring MVC framework which in my opinion is a very good thing as it allows other developers to improve or customize CAS according to their needs easier by just being able to understand these common used frameworks.

CAS has libraries for different authentication methods like authenticating user against LDAP or database. Also its very easy to configure your own authentication handler, for example an authentication handler which calls an external service.

As the registered services must also implement the CAS protocol there are client implementations in different programming languages like Java, C#, PHP or Perl.

When is CAS useful and when its not useful?

CAS is useful when you want to allow users to access multiple web applications by requiring them to authenticate only once instead of multiple times. Obviously you cant have multiple users with same username, this means that you might want to have only one place in which you store usernames and passwords.

One of the most problematic issue which I've found in CAS is that it does not allow you to group registered services which are accessible by a user after authenticating in CAS. For example if I have four registered services, after an user with a certain role logs in he should be able to access only first two of the four services and a second user with other privileges should be able to access only the last two of the services. From the short research which I've done there is no support for this. A way to solve this is to have two CAS instances running or another would be to customize CAS to work with something similar to ACL - Access Control List.

Useful resources

• Official website
• Github Jasig repository
• Apereo - CAS
• Jasig Central Authentication Service in 10 Minutes
• Install CAS
• CAS Demo
• Unicon CAS consultancy company

The end!

Even if I am not an expert in developing applications which implement CAS protocol or use CAS application as a solution for single-sign-on I wanted to share what I know and I hope that this will help other developers to get them started with CAS. I will also try to create a blogpost in which to describe a step by step and from scratch CAS installation and configuration. 

You're feedback and comments are more than welcome!

Good luck!

Robert Rusu

Book review: Thinking in Java - Fourth edition

To become a better programmer you should always look into reading new programming books, watch online tutorials or attend to related workshop. For Java programmers this book is one of the best and it provides detailed explanations and examples about the most important Java features.

I read this book once and later came back several times when I wanted to refresh my knowledge about a certain Java topic. As it has been such a useful learning resource for me I want to share it with you. To do so I composed a short and straight to the point book review.

Introduction

Thinking in Java was written by Bruce Eckel who also wrote other popular books like: Thinking in C# and Thinking in C++. The book was revised and improved multiple times, as a result of this multiple editions were published. The last one, fourth edition, was published in 2006.

In my opinion you at least must have little to medium programming background and prior hands-on experience Java/C#/C++. Besides this even if you are an experienced Java developer, if you did not read this book yet I highly recommend to do so as you will find a lot of information which will certainly improve your skills.

Good things

• The book is designed in such a manner that it helps you to learn fast starting with simple Java topics like Operators, Objects and Access Control to more advanced Java features like Annotations, Concurrency and Graphical User Interfaces.
• This book contains a lot of examples and exercises. 
• Interesting and relevant references are given when approaching a new topic.
• Explanations are easy to understand.
• Author mentions deprecated methods and why those should not be used anymore.
• Covers common pitfalls.

Bad things

• Code is not formatted and hard to read.
• After reading Clean Code: A Handbook of Agile Software Craftsmanship my standards about the quality of the code became higher than what this book has to offer.

Final note

Reading a technical book is not enough, to get most value of your time you should run all code samples, modify them to crash or even think how to improve them. Besides this you should also solve all exercises from the book in order to clearly understand the approached topics. So what are you still waiting? Open the book, start your favorite IDE and start learning!

Feel free to add your opinion in a comment about this blog post or about this great book.

Robert Rusu

Install APCu on Windows

Assumptions

-I assume that you know what is APC - Alternative PHP cache 

-You want to install APCu because APC is not compatible anymore with PHP 5.5.x

-You want to install APCu for wamp, xampp. Mostly windows web development platforms for PHP

Instructions

Pre: All directory locations might be different for you depending on your wamp installation folder and your PHP/apache versions.

1. Go to http://pecl.php.net/package/APCu, there is a table with available releases

2.Choose whatever release suits you better(I chose 4.0.5 DLL)  

3. Choose package from DLL list, depending on what Windows you are using(32 bits/64 bits) and PHP version. In my case I chose 5.5 Thread Safe (TS) x86

4. Unzip the archive, copy php_apcu.dll in C:\wamp\bin\php\php5.5.12\ext.

5. Go to C:\wamp\bin\apache\apache2.4.9\bin open php.ini  and add the following lines(I just added them at the end of the file):

[apcu]

extension="C:\wamp\bin\php\php5.5.12\ext\php_apcu.dll"

apc.enabled=1

apc.shm_size=32M

apc.ttl=7200

apc.enable_cli=1

apc.serializer=php

This are recommended configurations located in INSTALL file from the php_apcu archive, excepting the location of the DLL file.

6. Restart wamp

7. Go to http://localhost/phpinfo.php and check if apcu configuration table appears and apcu is enabled

8. If you also want to use apcu for PHP CLI then you only need to add in C:\wamp\bin\php\php5.5.12\bin\php.ini the config lines you added at step 5 in apache's php.ini.

The end!

Now you should be ready to start developing faster applications! I hope this helped everyone out there who did not find a tutorial on how to install APCu for windows. I also encourage you to leave me some feedback!

Robert Rusu

Things that you should do in order to become a better programmer

All of us try to become better at what we do, as a programmer I will try to give you a small list of things that will help you to become a better programmer.

#1 Always try to help others

Its always a good thing to help others when they are in trouble, when they need to learn new things or they just need an advice!

First of all helping others will help you understand better a concept or that something that they need explained. Your knowledge regarding that technology or whatever you help them with will get better just by trying to explain it. Being able to explain it in simple ways and giving others simple examples denotes the fact that you can master it.

For example helping others understand a design pattern, explaining them when and how to use it might very well refresh your memory regarding what you know about it. This would be another reason to help others. Helping others will make you feel better however its also important to not neglect your work!

#2 "Learning from the big fish"

This is a term that I often use when I am trying to say that you should always try to learn new things from more experienced programmers. For example I always try to see how more experience programmers act in different situations, how they solve different problems and what practices they use.

Its very important to also understand why more experienced programmers use a tool instead of another or why they are applying a design a pattern instead of another. Just copying what the "big fish" do is not enough!

"Learning from the big fish" can be accomplished in many other ways than meeting them face to face, I suggest reading books, watching videos or tutorials, watching(or even better going) to conferences hold by other programmers, reading articles or by just asking more experienced friends how to solve a particular problem or what would be a common solution to a general problem.

#3 Training

In my opinion its very important to train your programming skills. Even if you work 8 h a day I suggest that you should spend some time improving your skills. Programming is a skill and every skill is lost if you are not continuously training. 

This might be consisted of just working on a personal project in which you use another programming language than the one you use at work. Yes, learning a new programming language will improve your way you are programming in your favorite language just by understanding new concepts, existing problems solved in another manner than you were used to or just getting a fresh perspective.

It might be just you trying to understand concepts that you did not quite understand during work. It might be something new that you will encounter at work and you are trying to get a perspective about it. Or it can just be you write some small program to sharpen your coding speed or improve your analytic thinking.

Good luck!

Robert Rusu

Compile error Easy Mock

Easy Mock is a mocking library for Java which can help you to test your code isolated from some other components.

If you ever get the next compile error:
"expected (java.lang.void) in easymock cannot be applied to (void)"
It may be a result of you trying to call EasyMock.expect(mock.methodWhichReturnsVoid()), instead of doing that you should use expectLastCall method, you can use it like:
mock.methodWhichReturnsVoid(); 
EasyMock.expectLastCall();

I posted this tip because it took me several minutes to figure out what I was doing wrong, by the way, I was really tired!

Robert Rusu

Symfony2.3 form, grandchildren forms are not validated

Recently I had some issues with Symfony2.3 grandchildren forms not being validated even if I used as default option 'cascade_validation' => true. After doing some research I found other people having same issue and I found out that the solution is that besides putting 'cascade_validation' to true as default option to the form I also had to 'cascade_validation' => true in the collection of children and grandchildren forms I added.

Check the code sample below
Father form

class FatherType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder->add(
            'children',
            'collection',
            array(
                'type' => new ChildFormType(),
                'allow_add' => true,
                'allow_delete' => true,
                'by_reference' => false,
                'required' => false,
                'cascade_validation' => true //important to be added
            )
        );
    }

    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(
            array(
                'data_class' => 'Company/SomeBundle/Form/Entity/FatherFormEntity',
                'cascade_validation' => true, //important to be added
            )
        );
    }

    public function getName()
    {
        return 'father';
    }
}

Child form

class ChildType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder->add(
            'grandchildren',
            'collection',
            array(
                'type' => new GrandchildFormType(),
                'allow_add' => true,
                'allow_delete' => true,
                'by_reference' => false,
                'required' => false,
                'cascade_validation' => true //important to be added
            )
        );
    }

    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(
            array(
                'data_class' => 'Company/SomeBundle/Form/Entity/ChildFormEntity',
                'cascade_validation' => true, //important to be added
            )
        );
    }

    public function getName()
    {
        return 'child';
    }
}

Grandchild form

class GrandchildType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder->add(
            'attribute',
            'text'
        );
    }

    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(
            array(
                'data_class' => 'Company/SomeBundle/Form/Entity/GrandchildFormEntity',
                'cascade_validation' => true, 
            )
        );
    }

    public function getName()
    {
        return 'grandchild';
    }
}

References: https://github.com/symfony/symfony/issues/5204


I hope it helped you!
Robert Rusu

Writing clean code and following the standards

What does "writing clean code" and "following the standards" mean?

You might wonder what does "writing clean code" mean, well it means that as a programmer you write code that can be easy to read, understand, modify and test. Also what I mean by "following the standards" is that we should learn and apply the rules and conventions of a programming language, team and community.

For example Oracle offers the coding conventions for Java or Microsoft offers the coding conventions for C#. Another example would be that if you would like to contribute for developing a framework like Symfony2 you must also follow its conding standards.

Why should we write clean code and follow the standards?

In our days most of the real world applications need continuous development and even small programs require to be changed or improved after a while. Sometimes we often share pieces of code or entire applications to help others. Also from time to time we need to update or fix code we wrote a long time ago. As a result of all this we often spend our time reading code, understand what does it do and we will be required to modify it. That is why writing clean code will help us to deliver quality code which others can understand and can update it to their needs, we will be able to fix bugs and add new features or update the code faster.

Examples of why it's important to write clean code

Nowadays applications get bigger and are used by a lot of users everyday, many of them offer a lot of features and rarely you are the only programmer working on an application, often you will work in a team or even sometimes several teams work on an application, each working at one of its module. You will often need to read and fix a piece of code wrote by one of the other team members which is in vacation, imagine how hard it would be if the code would have been written in chaotically? Maybe you would think that it would have been faster to rewrite the whole thing again? But no investor is willing to pay you to rewrite a piece of functionality again, maybe you would not write it better than the previous team member. Now imagine how easy it would be for you to change the code if it was written clean and easy to understand, even more it was written by following the coding style and standards on which the team agreed at the beginning of the project? It would be a lot easier and you would not need to suffer just because you need to do a bug fix.

Another example of why we should write clean code and follow the standards which I've experienced was that I was assigned a job to update a relative small PHP script designed for interacting with twitter, even if I am experienced with PHP I was unable to understand what did the code do because the code was written chaotically, half of its naming was in the native language of the previous programmer, it was not in English which is the common language used for naming variables, methods, classes, folders etc. The files had between five hundred and one thousand lines of code, variables were not suggestive(at least ones which were named in English). It would have been easy for me to update the script if it was written clean, had meaningful names, if it had been divided logically into modules.

Resources

There are a lot of resources out there which will help you start writing clean code, one of my favorites is a book called Clean Code: A Handbook of Agile Software Craftsmanship written by Robert C. Martin in who is a promoter of writing clean code. 

For following the standards of a programming language in which you write code you should consult the official coding conventions, also if you use a framework you should check out what is the standard coding style on top of which it was built, in general it should be the same of it's related programming language. Besides that a framework often has a standard way to organize classes and other resources, this is one of the many advantages of using a framework, that other programmers can understand the code easier.

Important rules when writing tests for your application

Today I want to share some of the most important rules I follow when writing tests for an application. These are some rules I found useful to follow so that tests remain a part of my job which improves on the long term the quality of the code I deliver and makes my life easier. In my opinion same rules apply whether you are doing test driven development or just writing tests after you completed a feature, of course that its preferable to do test driven development as it brings more value to your code quality and your application.

1. Tests should be small! What does this mean? Well tests should be easy to read, easy to understand and the test should not have hundreds of lines of code. When writing tests you should follow the same rules as you do when writing clean code. Often after several days, weeks, months or even a year you will need to update a feature, which means the tests should be updated, but how will you update the test if it was written such that you cant understand it?
2. As you might know there are functional tests and unit tests, the purpose of a test should be to verify that a single functionality works in case of functional tests and in case of unit tests the purpose should be to verify that a method has the expected behavior which can be to return a result or to modify the state of an object.
3. Tests should be independent from each other! One or more tests should not fail because one other test fails. If a test failed it should be because the feature or the method has some issues in its implementation. Also the order of running the tests should not matter, running the tests should output the same result. An example of dependent tests is when first test creates a user and tests if it was saved in the database and then the second tests uses that saved user from database to test the login functionality.
4. Tests should be independent of the existing data in your database. What I mean by that is that as long as the database schema is the same every time you run your tests the same output should appear. For example you cant suppose when you test a login feature that the same username and password will exist in your database. In your test you should always either use mock objects or insert a user in the database, test login and then remove the user from database. Of course that most of the time its preferable to use mock objects instead of using the database. Same applies for resources used by the application, resources like log files or just simple files in which you store information. Usually when you have a really small application its not worth to use a database and you can use a text file to store information, so when writing a test make sure that in that test you also create a mock text file and delete it after your asserts finished.
5. Its very important how fast your tests are. It's a waste of time to wait dozens of minutes to run all tests for your application each time you finished a feature, sometimes you will need to run tests after each couple of minutes of writing code. I have seen very big applications(put in production but continuously improved during several years) for which running tests takes even few hours on a a high performance machine. In general to make your tests faster you can decouple your application from the database and use mock objects in your tests and you can not test the GUI of an application each time but that would mean that the "view" part of your application has very small amount of logic, which is a good practice. See how testing an application helps us to write better code? 

There a lot of other rules I might have missed, but consider this just a starting point!

Maintainability of the code by constant refactoring and unit testing

How often, as a programmer, you delivered new features in the beginning of the project so fast that clients could not believe it, but later, when the project got bigger, the number of lines of code grew and the number of features became larger, you started delivering features very slow? Often that was because the bad architecture of the application but mostly it was because the ugly code you were writing!

What did you do next? Well I suppose that you tried to refactor the code but you might have not succeeded because you were scared that you would introduce bugs. But keeping the code as it is wouldnt be a solution because as you know code rots as time passes, the code becomes uglier and uglier with every new feature or small change if you do not clean it. The solution would be to have unit tests, right?
If there would be tests and you would not be scared to do the refactoring of the code. But then you would need to convince your clients to spend money on refactoring working code. Most of them will not understand that refactoring will mean less money and less time spent on the long run.

I think that the best solution to this situation would be that whenever you work on a new feature or change an existing one because requirements change you should always leave the code cleaner that it was before. When see a piece of ugly written code apply one of the appropriate refactoring pattern without being scared that you might break the behaviour of the application because you are backed up by your unit tests.

Following this rule "leaving code cleaner that it was before you started working on it" and having unit tests will help you have a constant delivering time of new features, no delays because of the complexity of existing code, no bugs introduced by small changes and it will make you happier while working because you know you are delivering high quality software.

Top 3 Web Design Teaching Tips

Top 3 Web Design Teaching Tips

Thanks Stefan Mischook!

SymfonyLive Portland 2013 - Alexander Mols - Taking caching to the next ...

I recommend you to watch the video because Alexander presented a lot of tips and explained a lot of things regarding caching.
Topics discussed:

1. Caching for Routing, Container component of Symfony2
2. Caching in Twig
3. Doctrine Caching
4. Varnish
5. Tricks to prevent making a request to the Symfony2app for misc stuff like showing an html edit anchor