数据记录倡议DDI课件.ppt

1、International Workshop on Introduction to the DDI and the IHSN Microdata Management ToolkitBeijing,17-19 June 2013UNITED NATIONSDEPARTMENT OF ECONOMIC AND SOCIAL AFFAIRSSTATISTICS DIVISIONNATIONAL BUREAU OF STATISTICS OF CHINADDI元数据标准及IHSN国际住户调查网络微观数据管理工具国际培训班北京，北京，2013年年6月月17日日-19日日联合国经济和社会事务部统计司中华

2、人民共和国国家统计局Generic Statistical Business Process Model(GSBPM)Workshop objectives-ContextDescribes statistical processes(e.g.,implementation of a survey)in 9 phases,each divided into sub-processes.A convenient tool for assessment,planning of statistical processes.Specify the needsDesignBuildCollectProc

3、essAnalyzeDisseminateArchiveEvaluateMetadata ManagementQuality Management3通用统计业务流程模型(GSBPM)培训班目标 背景描述统计流程（例如，实施一项调查）的9个阶段，每个阶段有各自的子流程。一个用来评估与规划统计流程的便利工具。指明需求设计建立收集 处理分析传播存档评估元数据管理质量管理4The workshop will introduce standards and tools for:Workshop objectives Metadata management The DDI standard IHSN Me

4、tadata Editor Dissemination Policy,technical and ethical issues NADA software Archiving Preservation of digital informationSpecify the needsDesignBuildCollectProcessAnalyzeDisseminateArchiveEvaluateMetadata ManagementQuality Management5培训班介绍标准和工具的目的是：培训班目标 元数据管理元数据管理 DDI标准 IHSN元数据编辑软件 传播传播 政策，技术和道德问

5、题 NADA软件 存档存档 数字信息保存指明需求设计建立收集处理分析传播存档评估元数据管理质量管理6Part 1 Documenting your surveys and censuses using the DDI Metadata Standard and the IHSN Metadata Editor(Nesstar Publisher)Metadata management7第1部分 使用DDI元数据标准以及IHSN元数据编辑软件（Nesstar发布软件）记录您的调查和普查元数据管理8 To increase the credibility and transparency of t

6、heir statistical outputs To preserve institutional memory To allow replication of data collection and analysis To allow re-use or re-purposing of the metadataWhy do data producers need metadata?9 为了增加其统计输出的公信力公信力和透明度透明度 为了保持机构记忆机构记忆 为了允许复制允许复制数据收集和分析 为了允许重复使用或重新利用重复使用或重新利用元数据为何数据生产者需要元数据？10 To fully

7、 understand the(micro)data and make good use of themTo minimize the risk of misuse/misinterpretation,users need to fully understand the data.Why,by whom,when,and how data were collected and processed are important information.For making data discoverable in on-line catalogsUsers will know about the

8、availability of your data by searching or browsing detailed metadata catalogs.Why do data users need metadata?11 为了充分认识认识（微观）数据并很好的利用很好的利用他们为了尽量减少误用/曲解的风险，使用者需要充分了解数据。数据收集和处理的重要信息包括：目的，收集者/处理者，时间和方式。为了便于搜寻便于搜寻在线目录中的数据使用者通过搜索或浏览详细的元数据目录，将会知道是否可以获得您的数据。为何数据使用者需要元数据？12 The Data Documentation Initiative

9、(DDI)metadata standard helps structure,preserve and share survey or census metadata The IHSN Microdata Management Toolkit,a.k.a.Nesstar Publisher,provides a free and user friendly solution to document and catalog surveys/censuses in compliance with the DDI standard and international best practicesSt

10、andards and tools13 数据记录倡议数据记录倡议(DDI)元数据标准有助于结构化，保存和分享调查或普查的元数据 IHSN 国际住户调查网络微观数据管理工具包国际住户调查网络微观数据管理工具包，又名Nesstar发布软件，为记录并编目符合DDI元数据标准和国际最佳实践的调查/普查，提供了一个免费且用户友好的解决方案。标准和工具14 A checklist of what you need to know about a study and its dataset A structured and comprehensive list of hundreds of elements

11、 that may be used to document a survey dataset An XML metadata standard Developed by academic data centers/the DDI Alliance.Designed to encompass the kinds of data generated by surveys,censuses,administrative records.For microdata,not indicators.Two versions:Version 2.n(DDI codebook),used by the IHS

12、N Toolkit Version 3.n(DDI life cycle)What is the DDI?15 一张列有您所需要知道的，有关一个研究及其数据集信息的核对表 一张结构化的综合列表，包含数百个元素，可用来记录一项调查的数据集。一个XML格式的元数据标准 由学术数据中心/DDI联盟开发。旨在涵盖由调查，普查，行政记录产生的这类数据。用于微观数据，而非指标。两个版本：版本2.n(DDI码本),用于IHSN国际住户调查网络工具包 版本3.n(DDI生命周期)什么是DDI元数据标准？16 XML stands for eXtensible Markup Language.It is use

13、d to structure information to be shared on the Web or exchanged between software systems.XML is a file format,readable by any text editor(e.g.,Notepad).XML tags text for meaning.HTML tags text for appearance.The“tags”are conceptually the same as“fields”in a database.In an XML file,the information is

14、 wrapped between an opening tag and a closing tag.The tag name indicates its content.What is XML?17 XML代表可扩展标记语言可扩展标记语言，用于结构化在网络上共享或在软件系统之间交换的信息。XML是一种文件格式，在任何文本编辑器（例如：Notepad）上可读。XML语言的标签文本具有内容含义。HTML语言的标签文本用于文字外观。在XML语言下的数据库中，“标签”和“字段”在概念上是相同的。在一个XML文件中，信息被包裹在开始标签和结束标签之间。标签名称表示其内容。什么是XML？18“The Na

15、tional Statistics Office(NSO)of Popstan conducted the Multiple Indicators Cluster Survey(MICS)with the financial support of UNICEF.5,000 households,representing the overall population of the country,were randomly selected to participate in the survey,following a two-stage stratified sampling methodo

16、logy.4,900 of these households provided information.”In XML/DDI this would look like this:Multiple Indicator Cluster Survey 2005 MICS 2005 National Statistics Office(NSO)United Nations Children Fund Popstan National 5,000 households,stratified two stages 98 percent DDI and XML-An example19“Popstan国国

17、家统计局（NSO）在联合国儿童基金会（UNICEF）的资金支持下，开展了多指标类集调查（MICS）。调查采用二阶段分层抽样法，从参与这项调查的全国总人口中，随机抽取了5000户家庭作为代表全体的样本。其中4900户家庭提供了信息。”在XML/DDI中，以上内容呈现如下：多指标类集调查多指标类集调查 2005MICS 2005国家统计局国家统计局(NSO)联合国儿童基金会联合国儿童基金会Popstan国国全国全国5000户家庭户家庭,二阶段分层抽样二阶段分层抽样百分之百分之98DDI和XML-例子20 Can be transformed into many kinds of outputs:D

18、atabases,HTML,PDF,on-line catalogs,others Plain text files.Not specific to any operating system or application Easy to generate using specialized tools such as the IHSN Metadata EditorAdvantage of XML21 可以转化为多种输出：数据库、HTML、PDF、在线目录，及其他 纯文本文件，不是某个操作系统或应用程序的特定文件 使用特定工具生成非常便捷，例如IHSN国际住户调查网络元数据编辑软件XML的优势

19、22The DDI elements are organized in five sections:1.Document Description.Used to document the documentation process(“metadata on metadata”).2.Study Description.Information about the survey such as title,dates/method of data collection,sampling,funding,etc.3.Data File Description.Content,producer,ver

20、sion,etc.4.Variable Description.Literal question,universe,labels,derivation and imputation methods,etc.5.Other Material.Description of materials related to the study such as questionnaires,coding information,reports,interviewers manuals,data processing and analysis programs,etc.Structure of the DDI

21、2.0 standard23DDI元素由5部分组成：1.文档描述：用来记录文档著录过程（“元数据的元数据”）。2.研究描述：关于调查的信息，例如标题、数据收集的日期/方法、抽样、资金等等。3.数据文件描述：内容、生产者、版本等等。4.变量描述：字面问题、全域、标签、推导和估算方法，等等。5.其他相关信息：描述与研究相关的材料，例如问卷、编码信息、报告、面试官手册、数据处理和分析程序等等。DDI2.0标准的结构24Workshop participants will install the IHSN Metadata Editor(a.k.a.Nesstar Publisher)and docu

22、ment a small census dataset.Exercises25培训班与会者将安装IHSN国际住户调查网络元数据编辑软件（又名Nesstar发布软件）并学习记录一个小的普查数据集。练习26Content of the USB provided to participantsExercise data filesChinese version of:Popstan census data files(2)in Stata format Census questionnaire Enumerator manualSame content in EnglishSelected tech

23、nical and policy guidelinesIHSN Metadata Editor software and templates27USB存储盘向与会者提供以下内容练习的数据文件中文版本：Stata格式的人口普查数据（2个文件）人口普查问卷统计员手册英文内容相同技术和政策方面的指导原则IHSN国际住户调查网元数据编辑软件和模板28 Run NesstarPublisherInstaller_v4.0.9.exe to install the software Next step is to install the IHSN templates Exercise 1 Installa

24、tionOpen the Template Manager29 运行NesstarPublisherInstaller_v4.0.9.exe，安装软件 下一步是安装IHSN国际住户调查网络模板练习1-安装打开模板管理程序30Exercise 1 InstallationThen select the added template and click“Use”to activate it.This will now be the default study template.Click on“Import”and select the English(EN)or Chinese(CN)templ

25、ate found in folder“Software”Repeat the exact same process for the Resource Description Template31练习1-安装然后选择要添加的模板，点击“使用”来激活它。这个模板将成为默认的研究模板。点击“导入”，在“Software(软件)”文件夹中选择英语（EN）或中文（CN）模板 重复相同的步骤来添加资源描述模板32The next steps will be to document the Census:-Import the data files(Stata)-Add metadata in the D

26、ocument Description,Study Description,Data Files Description,and Variables Description sections-Attach and document the questionnaire and manual as external resources-Export the metadata to DDI(and RDF)formatsExercise 2-Documentation33接下来的步骤是记录普查：接下来的步骤是记录普查：-导入数据文件（Stata）-添加文件描述，研究描述，数据文件描述，和变量描述部分

27、的元数据-将调查问卷和面试官手册作为外部资源附加并记录-将元数据以DDI（和RDF）格式导出练习2 记录34When should data be documented?Much information loss,or never generatedDocument“as you go”not after completion of the operation.When documentation is done as a“last step”,much information is lost.35数据在何时应该被记录？“按进度”记录每一步 而不是在调查结束以后。如果只在“最后一步”记录数据，

28、许多信息已经丢失。36Available at www.ihsn.orgSoftware and guidelineshttp:/www.ihsn.org/home/node/117http:/www.ihsn.org/home/software/ddi-metadata-editor37可下载于可下载于www.ihsn.org软件和指导原则http:/www.ihsn.org/home/node/117http:/www.ihsn.org/home/software/ddi-metadata-editor38Part 2 Formulating a microdata disseminati

29、on policy,disseminating data and metadata,and the IHSN National Data Archive(NADA)softwareMetadata and microdata dissemination39第2部分 制定一个微观数据传播政策，数据和元数据的传播，以及IHSN国际住户调查网络国家数据归档（NADA）软件元数据和微观数据传播40 Diversity of research work.Data producers usually publish tabular and analytical outputs.But they will

30、never identify all the research questions that can be addressed using the data.Microdata dissemination encourages diversity(and quality)of analysis.Credibility/acceptability of data.Broader access to metadata and microdata demonstrates the producers confidence in the data,by making replication(or co

31、rrection)possible by independent parties.Benefits of dissemination41使研究工作多元化使研究工作多元化：数据生产者通常发布表格和分析输出。但他们绝不会辨识出这组数据能解决的所有研究问题。微观数据的传播促进了分析的多样性（和质量）。数据的公信力和认可度：数据的公信力和认可度：通过让独立的第三方能够复制（或修正）数据，对元数据和微观数据更广泛的访问显示了生产者对数据的信心。传播的优点42 Reduced duplication.Non accessibility to microdata forces users to conduc

32、t their own surveys.Microdata dissemination would reduce the risk of duplicated activities.It will also reduce the burden on respondents,and minimize the risk of inconsistent studies on a same topic.Funding.Better use of data means better return for survey sponsors,who will thus be more inclined to

33、support data collection activities.Quality of data.It is often through the use of data that insights for improvement for survey design can be identified.Benefits of dissemination43减少重复：减少重复：无法获得微观数据迫使用户自己进行调查。微观数据的传播将减少重复工作的风险。它也将减少受访者的负担，并将同一主题不一致研究的风险降到最低。资金：资金：更好地利用数据意味着对调查赞助者更好的回报，从而使他们更倾向于支持数据收

34、集活动。数据质量：数据质量：往往在数据使用的过程中，会产生如何改进调查设计的见解。传播的优点44 Exposure to criticism.Quality itself often puts a brake on microdata dissemination.Some data producers may fear to be exposed to criticism when data are not fully reliable,and to be confronted to the obligation to defend their results when challenged

35、by secondary users.Loss of exclusivity.When disseminating microdata,data owners lose their exclusive right to discoveries.This is more of an issue for academic researchers than official producers.Costs and risks of dissemination45受到批评：受到批评：质量本身往往会阻碍微观数据的传播。一些数据生产者可能担心当数据不是完全可靠时会受到批评，并且在面临二级用户质疑时，要承担

36、为自己的结果辩论的义务。丧失专用性：丧失专用性：微观数据的传播使数据拥有者失去了他们对自己发现的数据的专用权。相比官方数据生产者，这对学术研究者来说是更大的一个问题。传播的成本和风险46 Official vs.non-official results,and exposure to contradiction.Dissemination of microdata may lead to a proliferation of differing-and possibly contradictory-results and statistics.It may become more and mo

37、re difficult to distinguish between official figures and other sources of statistics.Financial cost.Properly documenting and disseminating microdata has a cost.This includes not only the costs of creating and documenting microdata files,but also the costs of creating access tools and safeguards,and

38、of supporting enquiries made by the research community.Costs and risks of dissemination47官方与非官方结果，对比揭露矛盾：官方与非官方结果，对比揭露矛盾：微观数据的传播可能激增不同的-并可能是相互矛盾的-结果和统计。传播可能导致官方数据和其他来源的统计数据变得越来越难以区分。财务成本：财务成本：妥善记录和传播微观数据是有成本的。这不仅包括创建和记录微观数据文件的成本，还包括建立访问工具和保障措施，以及向研究界提供辅助问询的成本。传播的成本和风险48 Confidentiality.One of the bi

39、ggest challenges of microdata dissemination is to minimize the risk of disclosure of any data that would compromise the identity of respondents.Legality.All countries have a specific national statistical and data protection legislation.Costs and risks of dissemination49保密性：保密性：微观数据传播的最大挑战之一，是如何尽量减少任

40、何由于披露数据而导致的，可能危及受访者身份保密性的风险。合法性：合法性：所有国家都有其特定的国家统计和数据保护法例。传播的成本和风险50 It is appropriate for microdata collected for official statistical purposes to be used for statistical analysis to support research as long as confidentiality is protected.Provision of microdata should be consistent with legal and

41、other necessary arrangements that ensure that confidentiality of the released microdata is protected.Principles-UNECEManaging Statistical Confidentiality and Microdata Access-Principles and guidelines of Good Practice,by the Conference of European Statisticians(CES)and United Nations Economic Commis

42、sion for Europe(UNECE)51 在确保保密性的前提下，研究者可以使用为了官方统计目而收集的微观数据，来进行统计分析并支持研究。提供微观数据应当符合法律和其他必要的约定，以确保被发布的微观数据的保密性。原则-UNECE联合国欧洲经济委员会管理统计保密性和微观数据访问管理统计保密性和微观数据访问-良好实践的原则和指良好实践的原则和指导方针导方针,欧洲统计学家会议(CES)与联合国欧洲经济委员会(UNECE)52 Statistical agencies are charged with protecting the confidentiality of survey respon

43、dents.Protecting confidentiality necessitates some sort of data anonymization so that individual respondents can not be identified.Anonymization53 统计机构被委以为调查受访者保密的责任。为了保密，必须采取一定的数据匿名化措施，从而使得个体受访者不会被辨识。匿名化54 Identifying variables include:Direct identifiers,which are variables such as names,addresses,

44、or identity card numbers.They should be removed from the published dataset.Indirect identifiers,which are characteristics whose combination could lead to the re-identification of respondents(e.g.,region,age,sex,occupation).Such variables are needed for statistical purposes,and should not be removed

45、from the published data files.Anonymizing the data involves determining which variables are potential identifiers and modifying the specificity of these variables to reduce the risk of re-identification to an acceptable level.The challenge is to maximize the security while minimizing the resulting i

46、nformation loss.Anonymization concepts55 识别变量包括：直接识别符直接识别符,是诸如姓名、地址或身份证号码的变量。这些变量应该从被公布的数据集中删除。间接识别符间接识别符,是一些个体特征变量，若组合在一起可重新识别受访者（例如地区、年龄、性别、职业）。这样的变量出于统计目的需要，不应该从被公布的数据文件中删除。数据匿名化涉及确定哪些变量是潜在识别符，并修改这些变量的特征，从而将重新识别的风险降低到一个可接受的水平。当前的挑战是如何在保持最大程度安全性的同时，最大限度地减少信息损失。匿名化概念56 Removing variables(e.g.,detai

47、led geographic identification)Removing records(outliers)Global recoding(e.g.,from age to age groups)Top-or bottom-coding(e.g.,create“65+”age category)Local suppression(replace with missing)Micro-aggregation(e.g.,for income variable)Data swapping Post-randomization Noise addition ResamplingAnonymizat

48、ion techniques57 删除变量（例如，详细的地理标识）删除记录（离群值）全球性重新编码（例如，将年龄改成年龄组）顶部或底部编码（例如，创建“65+”年龄组别）本地隐瞒（更换为缺失值）微聚集（例如，对于收入变量）数据替换 后随机化 添加噪声 重新抽样匿名化技术58Software:sdcMicroAn open source(R-based)packageAnonymization tools and guidelinesTechnical guidelineshttp:/www.ihsn.org/home/node/118More practical guidelines are

49、being produced by the IHSN.NOTE:Anonymization is a complex process.It requires analytical skills and involves some arbitrary decisions.59软件软件:sdcMicro一个开放资源的（以R语言为基础的）软件包匿名化工具和指导原则技术指引技术指引 http:/www.ihsn.org/home/node/118IHSN国际住户调查网络提供了更多的实际操作指引。注释:匿名化是一个复杂的过程。它需要分析技巧并涉及到一些主观的判定。60Formulating a micr

50、odata access policyPolicy guidelines on disseminationhttp:/www.ihsn.org/home/node/12061制定一个微观数据访问政策传播的政策指引http:/www.ihsn.org/home/node/12062 Data and metadata need to be made visible.Users will benefit from advanced data discovery tools,in particular on-line searchable catalogs.The IHSN developed an