Tips on Full stack development and Software Engineering

Posts

Lambda architecture

Lambda is a most discussed architecture pattern in data science space .The below is an attempt to explain it . Lambda is a data processing architecture and framework designed to address robustness and scalability and fault tolerance of big data systems. In this study we are planning to focus on the batch and speed layers to achieve data processing. As proposed in the summary for both the batch and stream we will be using Spark. What constitutes the lambda architecture for data processing? Lambda is a architecture pattern . Here the architecture we investigate has Lambda implemented primarily with Spark for batch and stream processing, Cassandra for No -SQL database storage, Kafka for accessing and sending the stream data and Zeppelin for visualizations. Sample application workflow investigated - In this sample application Spark is used for both batch and real time. We will persist the data in HDFS to perform batch operations against it. [HD

Picking a data set and trying to solve the problem.

Problem for us : Using Machine learning we will try to predict which valuable employees will leave next. And also create data explorations based on the dataset in hand.At large the application will be developed using R shiny and R for data analysis and presentation. At the data exploration phase, we are also considering using Python along with R so as to understand the differences in the platforms. Problem description : We intend to consider the HR survey results and understand why employees are leaving prematurely? Fields in the dataset include: · Satisfaction Level · Last evaluation · Number of projects · Average monthly hours · Time spent at the company · Whether they have had a work accident · Whether they have had a promotion in the last 5 years · Departments (column sales) · Salary · Whether the employee has left Dataset Links The data is from Kaggle https://

For Machine learning should we use R or Python ?

Some real important differences to consider when you are choosing R or Python over one another: Machine Learning has 2 phases. Model Building and Prediction phase. Typically, model building is performed as a batch process and predictions are done realtime . The model building process is a compute intensive process while the prediction happens in a jiffy. Therefore, performance of an algorithm in Python or R doesn't really affect the turn-around time of the user. Python 1, R 1. Production: The real difference between Python and R comes in being production ready. Python, as such is a full fledged programming language and many organisations use it in their production systems. R is a statistical programming software favoured by many academia and due to the rise in data science and availability of libraries and being open source, the industry has started using R. Many of these organisations have their production systems either in Java, C++, C#, Python etc. So, ideally they w

Scrumban using Asana

I had to create a project management practice in Agile for one of my projects. I had experience in working with Scrum and Scrumban Agile methodolgies . I decided to adopt Scrumban as i found it better that Scrum and more suited for product development with uncertain requirements and often user stories which require a depeer research and understanding . You can read on further from the links below on Scrumban https://www.agilealliance.org/what-is-scrumban/ http://leansoftwareengineering.com/ksse/scrum-ban/ I had to choose a tool that was open source and readily available , I evaluated the below tools Taiga Trello Asana Of the above I found Asana highly suitable and configurable and bingo they also had a mobile application which was quite user friendly. The Sprints were planned for 2 weeks duration and a Milestone of 1 month duration . Each Milestone had two Sprints . For scrumban I choose the below project structure Here the Backlog had the backlog item