Understanding the Importance of Feature Engineering in Data Science
Feature engineering, a manual and time-consuming process, is a crucial step in data science workflows. It involves generating and transforming features based on domain knowledge. Avoiding the pitfalls of past technologies like expert systems, feature selection plays a key role in determining which features to include in datasets. Competitive intelligence (CI) is defined as the function responsible for early risk and opportunity identification in the business environment. Utilizing models like CVI and IVC helps in gathering and analyzing information to make informed business decisions.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Feature engineering Prof. Amos DAVID Prof. Amos DAVID, DSA Abuja, 2018
Early warning Be careful not to remake what happened with expert systems in the 1990s where greate hope was placed on the concept Prof. Amos DAVID, DSA Abuja, 2018
Scientific study of CI "The scientist is not the man who provides the real answers. He is the one who asks the right questions" - Claude Levi-Strauss Prof. Amos DAVID, DSA Abuja, 2018
A definition of Feature Engineering Feature generation and transformation, called feature engineering is largely manual and often the most time consuming step in a data science workflow. It is a complex exercise, performed in an iterative manner with trial and error, and driven by domain knowledge developed over time Cognito: Automated Feature Engineering for Supervised Learning, Udayan Khurana, Deepak Turaga, Horst Samulowitz, Srinivasan Parthasrathy IBM TJ Watson Research Center 2016 IEEE 16th International Conference on Data Mining Workshops Prof. Amos DAVID, DSA Abuja, 2018
Feature selection Feature selection: The task of selecting which features to include in our dataset, out of all the possible features that we could have considered. Prof. Amos DAVID, DSA Abuja, 2018
Definition of CI A more focused definition of CI regards it as the organizational function responsible for the early identification of risks and opportunities in the market before they become obvious. Experts also call this process the early signal analysis. Key points of this definition: Competitive intelligence is an ethical and legal business practice, as opposed to industrial espionage which is illegal. The focus is on the external business environment There is a process involved in gathering information, converting it into intelligence and then utilizing this in business decision making. Some CI professionals erroneously emphasize that if the intelligence gathered is not usable (or actionable) then it is not intelligence. Prof. Amos DAVID, DSA Abuja, 2018
CVI Model Identify the possible use of the information for solving the decisional problem Verify the relevance of information sources and potential information V I C Collect information Prof. Amos DAVID, DSA Abuja, 2018
IVC Model Identify the possible use of the information for solving the decisional problem Verify the relevance of information sources and potential information I V C Collect information Prof. Amos DAVID, DSA Abuja, 2018
CI viewed as a process Identification of needs in the form of problems to solve or stakes (threat, risk, danger), 1. Identification of the types of necessary information to obtain the result, 2. Identification and validation of the relevant information sources, 3. Collection of information, 4. Validation of the information collected, 5. Processing of the information collected for the calculation of indicators, 6. Interpretation of the indicators, 7. Decision making for the resolution of the identified problem. 8. Prof. Amos DAVID, DSA Abuja, 2018
Early spotting of opportunities : From 1990S IT Penetration through mobile telecommunication in Nigeria Prof. Amos DAVID, DSA Abuja, 2018
Extract n 1 Vodacom Nigeria says potential for 40 million mobile users Friday 16 January 2004 | 09:55 CET | News Vodacom believes that the potential number of mobile phone subscribers in Nigeria could be double the current estimate of 20 million users. As a result, up to four operators would be able to compete within the market. According to Alan Knott-Craig, the Nigeria market is big enough to support both Vodacom and MTN (S Africa), mobile operator which entered the Nigerian market in 2001. Vodacom believes that Nigeria has the potential to be as big as S Africa, where the market potential is 25 million-30 million users by 2007- 2009, compared to 17 million users at present. The biggest obstacle to Vodacom establishing itself in Nigeria is strict US anti-corruption legislation, which Telkom and SBC Communications must comply with. http://www.telecompaper.com/news/vodacom-nigeria-says-potential-for-40-million- mobile-users, 2004-01-16 Prof. Amos DAVID, DSA Abuja, 2018
Extract n 2 Nigerian telecom market to hit $10b in 2010 Written by Henry Umoru Thursday, 30 October 2008 Federal government has projected that the market value of the nation s telecommunication industry would hit a total of $10 billion by 2010. Speaking at the opening of the 5th edition of Bridges Across Borders (BAB) conference held at the NICON Luxury Hotel, Abuja, Minister of Commerce and Industry, Engr. Charles Ugwuh who noted that the current 50 million telecom subscriber base with the current value put at $5 billion per annum, said the Nigerian average spending on telephone service is about N2, 570 per month. The Conference is being organized by the Nigerian Export Promotion Council (NEPC) and it participated in the event for the first time in Johannesburg, South Africa in 2006 with other 13 countries in attendance. According to him, there is the tendency that the present 50 million subscribers will keep growing as new investors enter into the sector, just as he stressed that the liberalization of the telecommunication industry in 2001 has brought about new investment in the business environment. http://allafrica.com/stories/200810300123.html, 2008-10-30 Prof. Amos DAVID, DSA Abuja, 2018
Extract n 3 Tuesday, August 28, 2012 Nigerian smartphone market to grow by 35% -Huawei by Dayo Oketola About 35 per cent increase is expected in sales of smartphones in Nigeria before 2015. The Managing Director, Huawei Devices Nigeria, Mr. Tony Liangwei, said this would translate into 30 million smartphones expected to be sold in the country in the next three years. Speaking during a workshop for mobile device dealers in Lagos on Friday, Liangwei said the expected increase in sales would be driven by the shift in the focus of Nigerian consumers from phones with basic functionality to smartphones. This, according to him, means that in the years to come, the opportunities in the smartphone market in the country for device dealers will be much greater than what is available today. In view of this, he said it was essential for dealers to partner with Huawei as the company was set to introduce devices that would revolutionise the mobile and Internet technology market at unbeatable prices. Liangwei said Huawei organised the workshop to formally introduce its devices as well as to educate dealers on the background, core values and organisational culture of the phone maker. He further explained that the technology company used the event as an opportunity to intimate dealers in Nigeria with its growth agenda. We want to use the workshop as a platform to encourage dealers in mobile devices to form partnerships with Huawei, which will be beneficial to all parties, he said. In order for dealers to key into the opportunities available in the Nigerian smartphone market, Liangwei said it was important for them to consider partnering with Huawei. The three major product lines highlighted at the seminar were Huawei's handsets, mobile broadband and home devices. Industry analysts have said Huawei devices should be among the most affordable considering their technology and quality. The company produces a wide array of devices ranging from low-end mobile devices to high-end smartphones. Prof. Amos DAVID, DSA Abuja, 2018 Liangwei said he was looking forward to a mutually rewarding partnership with the dealers and the country as a whole.
Extract n 4 Nigeria's monthly phone calls to hit N106bn - Investigation Monday, September 24, 2012 By BISI OLALEYE Nigeria's telecommunications subscribers monthly expenditure on mobile phone calls, according to checks, would soon hit a whopping N106.1 billion by the end of December, 2012. The call expenditure forecast is a conservative calculation based on the current industry Average Revenue Per User (ARPU) and the projection by experts in the industry that stipulated that active telecoms subscribers would have grown higher than what the telecoms companies currently have on their respective networks. Former Executive Vice Chairman, Nigerian Communications Commission (NCC), Ernest Ndukwe, has predicted during the Nigerian Telecoms Awards, organized by Logica Group recently, that active telephone subscriptions in the country would surpass 105 million in the next three months, including December 2012. However, according to the latest statistics for the month of July 2012, released by the NCC, there are about 103 million telecoms subscribers currently on all the telecoms networks in the country. While the industry ARPU in Nigeria was estimated at around N1,011, according to the Business Monitor International Limited (BMI), and subscriber base of 103 million in the month of July, the outgoings by Nigerian subscribers in July was conservatively valued at N104 billion monthly. ARPU is the financial benchmark used globally by telecoms companies to measure the average monthly or yearly revenue generated from an average subscriber. The expenditure increased from N100 billion in January 2012 when active industry subscriber base was estimated at 99 million by the NCC, to reach N104 billion in seven months after into the month of July when subscriber base hit 103 million. Prof. Amos DAVID, DSA Abuja, 2018
Extract n 4 With the industry projection by Ndukwe on the industry records,telcos will have 105 million active telecoms subscribers by the end of December, 2012, and with industry ARPU of N1,011, Nigerians subscribers are billed to spend an average of N106 billion monthly. The projected expenditure is also equivalent to the average monthly revenue from phone calls, which will accrue to the telecoms firms, including the Global System for Mobile Communications networks such as MTN, Globacom, Airtel, Etisalat; the Code Division Multiple Access operators of Visafone, Capcom (MultiLinks , Starcoms), dormant Zoom Mobile, as well as the fixed line operators. Elaborating on the boom in subscriber growth since 2011 telecoms deregulation that was undertaken by the Federal Government, Ndukwe said no one was in a position to predict in those early days of GSM licensing, the full potential of the market and the speed at which the Nigerian telecom network would grow. However, he stressed that, "Today, the figure for active subscribers in the mobile networks is around 100 million lines and is likely to surpass 105 million by end of December 2012. "Nigeria has transited from what I described as the telecommunications dark ages before 2000 to a telecommunication revolution age that has opened up new possibilities and frontiers across our political social and economic landscape." Prof. Amos DAVID, DSA Abuja, 2018
Extract n 5 Friday, October 5, 2012 Active mobile telephone lines now 105.2 million by Dayo Oketola The country's mobile telephone subscriber base has continued to record strong growth with the number of active mobile telephone lines hitting 105.2 million in August, the latest subscriber data released by the Nigerian Communications Commission has revealed. The data, which was released on Thursday, revealed that mobile network operators in the country added over 1.8 million new lines in August alone thereby, increasing the subscriber base from 103.4 million active telephone lines in July to over 105.2 million at the end of August 2012. Industry analysts said the promotions currently being run by Global System of Mobile telecommunications operators in the country might have been responsible for the huge subscriber growth in August. While the total connected lines increase to 141.2 million in the month, the NCC data showed that GSM companies such as MTN, Globacom, Airtel and Etisalat attracted most of the new subscriptions between January and August, while the Code Division Multiple Access operators such as Visafone, Multi-Links, Starcomms and ZoomMobile continued to witness month-on-month decline in combined subscriber base. According to the NCC data, while active mobile subscriptions on the GSM networks increased from 91 million in January to 101.4 million at the end of August; the CDMA operators' subscriber base shrank from 4.4 million to 3.3 million during the eight-month period. The GSM active subscriptions, which stood at 91 million in January, 2012, increased to 92 million in February and 94.5 million at the end of March. The subscriptions continued the upward growth trend, reaching 96.6 million in April and 97.5 million in May. In June, July and August, the combined active GSM subsriber base increased to 98.3 million; 99.4 million and 101.4 million respectively. However, the active subscriber base on CDMA networks continued to decline from 4,031,820 lines in February to 3,904,846 in April. It further plunged to 3,718,153 in May; 3,541,355 in June; and fell in July to 3,452,368 and finally to 3,347,716 at the end of August 2012. The number of fixed lines, which stood at 688,333 as at January, declined to 488,088 at the end of August 2012. Prof. Amos DAVID, DSA Abuja, 2018
Extract n 5 While the Nigerian telecoms industry continues to experience huge growth year-in-year, it has once again defied industry forecast, with the giant leap it recorded in growth in August. The former Executive Vice-Chairman, Nigerian Communications Commission, Mr. Ernest Ndukwe, had recently predicted that the country's mobile subscriber base would hit 105 million in December 2012. However, the latest NCC data revealed that the target was achieved much earlier, as active subscriptions had crossed the 105 million projections, reaching over 105.2 million at end of August, 2012. Experts said though it was good that the country's mobile subscriber base had crossed the 105 million mark, the implication was that it would put pressure on the mobile networks if the operators did not increase their network capacities. It will be recalled that the NCC had recently fined the four GSM companies in the country to the tune of N1.17bn for poor quality of service on their networks. With increased subscriber base, analysts, urged the operators to match this with more investments in network expansion. Prof. Amos DAVID, DSA Abuja, 2018
Extract n 6 Thursday, February 14, 2013 Subscribers spent N2.14tn on phone calls, others in 24 months by Dayo Oketola Active telephone subscribers in the country must have spent a whopping sum of N2.14tn on voice calls and Short Message Service, among other basic services between January, 2011 and December, 2012. This is based on an Average Revenue per User of N912 monthly accruing from telecoms subscribers to MTN, Globacom, Airtel, Etisalat, Visafone, Starcomms, and Multi-Links, among other operators in the country. The ARPU is a financial performance benchmark in the telecoms industry that measures the average monthly or yearly revenue generated by Global System for Mobile communications, Code Division Multiple Access and the fixed telephone operators in a particular country. Nigeria currently has 113 million active subscribers. In view of this, telephone service subscribers, who increased from the highest of 95,886,714 in December 2011 to 113,195,951 in December 2012, would have spent N2.14tn on mobile services, particularly voice calls between January 2011 and December 2012. While a little over N1tn was spent on telephone services in 2011, subscribers spent 1.14tn on the services in 2012. Prof. Amos DAVID, DSA Abuja, 2018
Extract n 6 The Nigerian Communications Commission had revealed that active telephone users in the country stood at 89,840,343 in January 2011 and at a monthly ARPU of N912, at least N81.9bn was expended on phone calls. This increased to N82.6bn in February 2011 and N82.9bn in March, when the subscriber base hit 90,583,306 and 90,969,794, respectively. Subscribers further spent N82.6bn in April of the same year; N82.5bn in May; N82.6bn in June; N82.8bn in July; N83.9bn in August; N85.2bn in September; N85.6bn in October; N87.4bn in November and N87.4bn in December 2011. At that time, the subscriber base had increased to 90,834,429; 92,094,200; 93,461,436; 93,924,116 and 95,387,893 in July, August, September, October and November; closing at 95,886,714 in December. Therefore, over N1tn was spent on telephone services in 2011. In 2012, the total estimated telephone service spending was N1.14tn as at December. Specifically, 96,150,836 active subscribers spent an estimated N87.7bn on telephone services in January. This was followed by N88.1bn in February; N90.4bn in March; and N92.2bn in April, when the subscriber base was 96,616,580; 99, 145,013; and 101,077,658, respectively. Subscribers' telephone spending continued on an upward trend in May with N92.9bn by 101,814,533 active subscribers. This was followed by N93.4bn in June; N94.3bn in July; N96bn in August; and N97.7bn in September. By then, subscriber base had risen from 102,369,999 in June to 107,083,036 in September 2012. While N99.9bn was spent on telephone services in October, over 110 million subscribers expended N100.6bn in November and N103.2bn was spent in December 2012 by 113,195,951 telephone subscribers. Prof. Amos DAVID, DSA Abuja, 2018
Synthesis Evolution nigerian mobile subscribers 213,000,000 205,000,000 50,000,000 20,000,000 2004 2008 2012 January 2013 Prof. Amos DAVID, DSA Abuja, 2018
Statistics from NCC (lack of data 2001-2006) Active mobile lines 120,000,000 109,829,223 100,000,000 90,566,238 81,195,684 80,000,000 65,533,875 56,935,985 60,000,000 40,011,296 40,000,000 20,000,000 0 2007 2008 2009 2010 2011 2012 Prof. Amos DAVID, DSA Abuja, 2018
Early spotting of opportunities / 2 Prof. Amos DAVID, DSA Abuja, 2018
PUNCH January 8, 2013 by Dayo Oketola 25% of Nigerian mobile subscribers use smartphones TNS Only 25 per cent of Nigeria s over 105 million mobile telephone subscribers use smartphones, global market research firm, TNS, has said in a report entitled, Navigating growth in Africa. The report, which was authored by the Chief Executive Officer, TNS Africa and Middle East, Mr. Kim Macllwaine, and 12 other market research experts, including Mr. Aggrey Maposa, Melissa Baker and Mr. Nick Hills, revealed that Nigeria was the second biggest smarphone market in Africa, closely following South Africa. In spite of the country s smartphone penetration, the report revealed that 59 per cent of Nigerian phone users were still using basic feature phones, while only 16 per cent used advance feature phones. Prof. Amos DAVID, DSA Abuja, 2018
Conference initiative in West Africa Creation of ISKO-West Africa International society for knowledge organization Bi-annual conference on the theme Transition from Observation to Knowledge to Intelligence Since 2014 Prof. Amos DAVID, DSA Abuja, 2018
Thnaks for your attention Prof. Amos DAVID, DSA Abuja, 2018
