Data Integration brings the connectors that bring external data into the Common Data Service (CDS) and store the data in Common Data Model (CDM) form. These gateways make it easy for businesses to integrate their data in a single place.
Data Integration brings direct communication to hundreds of business-critical data sources, including:
- Hybrid connectivity via the on-premises data gateway.
- Integration into Power BI, PowerApps, Logic Apps, Microsoft Flow, Azure Analysis Services and Office 365 (Power Query in Excel).
Simpler Integration of Enterprise Data with Power Query and Data Integration Platform
By introducing new Power Query data preparation capabilities, enterprise-grade connectors and improvements for Common Data Service data integration, Microsoft has simplified common data integration issues.
Power Query Capabilities
Power Query is being enhanced with market-leading Smart Data Preparation experiences, including data extraction from semi-structured sources like PDF files, HTML pages, fuzzy-matching algorithms to rationalize and normalize data based on similarity patterns and data profiling capabilities.
By introducing these abilities, companies will find it easier to identify errors and outliers in the Power Query Editor.
Intelligent transforms and AI support
With single-click access to AI-based transforms, business analysts can easily incorporate AI-drive insights. Capabilities will slowly be introduced to this area starting with sentiment analysis and keyword extraction from natural language text. Capabilities such as OCR and image analysis will be added over time.
Analysts and BI Pros will gain access to new functions and APIs for AI transformations; this helps column transformations for sentiment analysis and keyword extraction with a single-click or single line of script.
The APIs will also be supported for use during installation and configuration of Power BI applications.
IntelliSense support for the M formula language
M language editing experiences are now being supported with IntelliSense support which makes users able to be more successful in editing their M code directly, find errors, discover M library functions and understand what parameters are needed.
Flow support for data filtering and mashup
Power Query Online can now be integrated with Microsoft Flow which will allow users to enable data filtering and mashup as part of the “Get Rows” action on specific connectors, such as SQL server.
Power Query documentation and community website
In Microsoft Docs you can find Power Query; these documents include samples, tutorials, how tos and a developer reference for building custom connectors for Power Query.
Also, a new Power Query community website has been launched. Included in this website is a technology overview, forums, UserVoice and community-driven content. This community focuses on the core of Power Query capabilities that enable partners to build new gateways and data transformations on the top of the platform.
Enterprise-grade connectors
Enhancements to Microsoft’s SAP Business Warehouse connectors (Application Server and Message Server) put Microsoft Power BI ahead of any other third-party BI vendors in terms of connectivity to SAP Business Warehouse (BW).
SAP has certified Microsoft’s connectors for SAP HANA and SAP BW. Also, large enterprise capabilities such as Security Assertion Markup Language (SAML) based single sign-on and SSL certificate validation, can currently be enabled through the improvement of the SAP HANA connector. Significant enhancement of this provides orders of better performance as well as additional capabilities.
Single sign-on support for Spark on-premises and releasing existing connectors are other connector improvements.
Improvements to connectors within PowerApps, Microsoft Flow and Logic Apps have also been introduced such as releasing write capabilities for Azure SQL Data Warehouse, numerous Visual Studio Team Services updates.
Evolvement of the platform continues and has added new connectors to work consistently across Common Data Service for Apps, Power BI, PowerApps, Microsoft Flow and Logic Apps.
Improved connectors and import experiences for CDS data integration
Microsoft will continue expanding support for data connectors and data transformations in the web-based Power Query experience. As part of the preview of Common Data Service for Apps that Microsoft unveiled in early 2018, there were built-in capabilities for business users. Users can import data from a wide range of data sources from Microsoft and third parties in the cloud and on-premises.
Custom connectors will also be added in Power Query Online, so existing connectors built by partners can be leverage for data integration with Common Data Services for Apps and Power BI dataflows.
The user experience for importing data from Power Query files is also improved, allowing users to upload local files.
Certified custom connectors in Power BI Desktop
Easier Sharing, Unification and Enrichment of Data with Common Data Model Entities’
The Common Data Model of standard entities has been improved for the integration of business processes, applications, and profile enrichment across sales, services, marketing, operations, finance, talent and commerce business functions. This also includes the core entity profiles for customers to support observational data enrichment and analysis.
Within this model, there is a large and growing collection of solutions that work efficiently together when data is stored in the Common Data Model form, which means a user can quickly implement new business processes, analytical solutions and gain a full view into businesses operations without complexity. Since the March 2018 release, the public Common Data Model GitHub repository has grown significantly to several hundred well-defined, modular and expanded business entities. This includes major extensions to support additional vertical industries in close collaboration with strategic partners.
Microsoft has been collaborating with an ecosystem of ISVs to digitally transform their business processes by creating industry extensions to increase range and flexibility of the Common Data Model. The interactive and graphical entity explorer allows users to better understand and control existing Common Data Model entities for businesses applications and analytical solution development.
The Common Data Model will have improved documentation and presentation content for customers and partners to better understand what’s included. Additionally, the model will demonstrate how to better use it within the platform and how to customize in a way that follows the model’s best practices. Discussion forums will also be added to support a community for application, data model and data integration template developers.
Partners will be able to find guidance on how to become a strategic partner for potential Common Data Model enhancements for industry verticals to bring more applications to the model platforms. This will improve the ecosystem of ISVs and app developers.
In addition, a built-in automatic support for the Common Data Model will include, the Power Query data integration experiences, data integration templates, CDS for Apps and Power BI dataflows. Data integration templates hasten integrating data into well-known Common Data Model entity shapes from common line-of-business applications and SaaS services.
Richer and More Unified Connector Developer Ecosystem
A major part of Data Integration and the suite of products is that it supports connectivity to external data sources. With the new release, an integrated experience will be provided to offer improved consumption experiences for connectors in Power BI Desktop. Certified connectors built by partners and ISVs will be easily accessed and use in the Power BI Desktop. This integration will improve the experience for our developers and consumers in the ecosystem. Both personal and enterprise versions of the on-premises Data Gateway will support discovery and data restores for certified and custom connectors.
Developers and ISVs building connectors may submit their connectors for certification to Microsoft. This unified connector certification program for connectors are released publicly and provide an integrated experience for end users. The results – reach, discoverability and usage of the service.
Enhanced Administrator Capabilities for Data Gateway and Data Integration
GDPR requirements will be of top priority this season. The data integrator for admins meets GDPR requirements that allow access and remove any other GDPR-granted data subject rights.
With the release of the data integrator for admins, most customers have control to customize templates for process-base integration. As every business is different, the experience will be improved by provided a way to create new templates from scratch and provide an ability to validate source and destination solutions.
In addition, the new Business Platform Admin Center will centralize the setup experience wherein customers need to switch between multiple portals. This includes a new feature of creating data integration support tickets in the same portal and project execution history that will provide full history over past executions.
Data Integration Templates improve getting data to the Common Data Service from common third-party systems of records and SaaS services. With Microsoft providing additional data integration templates, customers and ISVs can leverage the platform to build their own new templates to share them across CDS instance’s. The templates can be repackaged and reuse the customized templates without having to repeat the process.
With the assistance of Common Data Service for Apps role-based access, business users and admins will be able to assign detailed permissions that enable users within their organization to read, contribute to or own templates. These templates will also be easily discoverable in a central marketplace where customers can easily browse and obtain the latest templates.
Enterprise Grade Hybrid Connectivity Using the On-Premises Data Gateway
Earlier this year, custom connectors in Power BI Desktop was announced to influence the powerful capabilities of the M language that enabled partners to write their own connectors and allocate them to every Power BI user. These custom connectors stay up-to-date on the Power BI service by refreshing the data through the on-premises data gateway.
In addition, the high availability capabilities will include multiple experience improvements, around error reporting and improved user experience. SSO implementation will be easier to test the connection and diagnose issues with support of multiple domains. SAML-based single sign-on for supported data sources will now be added to the single sign-on scenarios.
Improved data source settings for managing the “Manage Gateways” page in Power BI will include skipping the test connection step, renaming data sources and creating multiple data sources with different credentials. Lastly, the ability to split requests traffic for a given gateway group across all gateways in that specific group. The gateway administrator will be able to turn this ability on and off according to their needs.
